Jul 10 19:05:29.200808 ip-10-0-132-221 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 19:05:29.200822 ip-10-0-132-221 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 19:05:29.200831 ip-10-0-132-221 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 19:05:29.201158 ip-10-0-132-221 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 19:05:40.516339 ip-10-0-132-221 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 19:05:40.516354 ip-10-0-132-221 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 366a16e11a9f4407986d622809db7d6f -- Jul 10 19:07:39.828910 ip-10-0-132-221 systemd[1]: Starting Kubernetes Kubelet... Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.249708 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255166 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255180 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255183 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255186 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255189 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255193 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255195 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:07:40.291407 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255198 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255201 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255204 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255206 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255209 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255213 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255215 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255218 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255221 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255224 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255226 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255229 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255232 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255235 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255238 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255241 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255243 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255246 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255249 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:07:40.293519 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255254 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255257 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255259 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255262 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255264 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255267 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255269 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255272 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255275 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255277 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255280 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255282 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255285 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255287 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255290 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255292 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255295 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255298 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255300 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:07:40.295263 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255303 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255306 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255309 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255312 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255314 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255318 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255321 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255324 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255327 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255329 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255332 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255335 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255337 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255340 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255342 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255345 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255347 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255350 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255353 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255357 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:07:40.296130 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255361 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255363 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255366 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255369 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255372 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255375 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255377 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255380 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255383 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255386 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255388 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255392 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255395 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255398 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255400 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255403 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255405 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255408 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255412 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:07:40.298122 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255415 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255418 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255422 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255425 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255427 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255430 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255432 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255435 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255437 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255440 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255443 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255445 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255447 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255450 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.255454 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256148 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256155 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256158 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256161 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:07:40.300713 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256164 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256166 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256169 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256172 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256175 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256178 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256180 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256183 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256185 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256188 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256190 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256193 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256196 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256200 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256203 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256206 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256208 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256211 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256214 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256216 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:07:40.303253 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256219 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256221 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256224 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256226 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256229 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256231 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256236 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256240 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256243 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256245 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256248 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256251 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256254 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256257 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256260 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256263 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256266 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256269 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:07:40.305465 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256271 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256275 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256278 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256281 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256284 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256286 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256289 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256292 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256295 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256299 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256302 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256305 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256308 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256310 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256314 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256317 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256320 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256322 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256325 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:07:40.307202 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256327 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256330 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256333 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256335 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256338 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256340 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256343 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256345 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256348 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256351 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256354 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256356 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256359 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256361 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256364 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256367 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256370 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256372 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256375 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256395 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:07:40.308127 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256399 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256403 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256407 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256410 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256413 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256416 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256418 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256421 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256424 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256427 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256429 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256432 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256434 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256437 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256439 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256447 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256449 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256452 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256821 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256827 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256829 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:07:40.308932 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256832 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256837 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256841 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256844 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256847 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256850 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256852 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256855 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256858 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256860 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256862 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256865 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256867 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256870 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256875 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256878 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256880 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256883 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256885 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:07:40.309758 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256888 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256890 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256892 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256895 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256897 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256900 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256903 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256907 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256909 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256912 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256915 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256917 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256920 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256922 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256925 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256928 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256930 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256933 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256935 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256938 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:07:40.310550 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256940 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256943 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256945 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256948 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256950 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256953 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256960 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256964 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256967 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256970 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256972 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256975 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256977 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256979 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256982 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256984 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256987 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256990 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256992 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:07:40.311357 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256996 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.256999 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257001 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257004 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257007 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257009 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257012 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257014 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257017 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257019 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257022 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257024 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257027 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257029 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257032 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257034 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257037 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257039 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257042 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257044 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:07:40.312158 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257050 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257053 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257056 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257058 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257062 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257065 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257068 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257070 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257073 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257075 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257078 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257081 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257084 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257088 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257091 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257094 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257097 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257100 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257176 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257186 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 19:07:40.312990 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257193 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257199 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257205 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257210 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257214 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257218 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257221 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257224 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257228 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257231 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257234 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257237 2579 flags.go:64] FLAG: --cgroup-root="" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257242 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257245 2579 flags.go:64] FLAG: --client-ca-file="" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257249 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257252 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257256 2579 flags.go:64] FLAG: --cluster-domain="" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257259 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257262 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257265 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257269 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257272 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257275 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 19:07:40.313812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257278 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257283 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257286 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257289 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257292 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257295 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257298 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257302 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257305 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257307 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257310 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257313 2579 flags.go:64] FLAG: --enable-server="true" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257316 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257321 2579 flags.go:64] FLAG: --event-burst="100" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257324 2579 flags.go:64] FLAG: --event-qps="50" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257327 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257330 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257332 2579 flags.go:64] FLAG: --eviction-hard="" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257336 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257339 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257342 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257346 2579 flags.go:64] FLAG: --eviction-soft="" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257349 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257352 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257355 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 19:07:40.314610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257358 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257361 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257363 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257366 2579 flags.go:64] FLAG: --feature-gates="" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257370 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257373 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257376 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257379 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257383 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257386 2579 flags.go:64] FLAG: --help="false" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257389 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-221.ec2.internal" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257392 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257395 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257398 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257401 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257404 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257407 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257410 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257413 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257415 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257418 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257422 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257424 2579 flags.go:64] FLAG: --kube-reserved="" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257427 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 19:07:40.315418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257430 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257433 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257436 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257438 2579 flags.go:64] FLAG: --lock-file="" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257442 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257445 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257448 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257453 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257457 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257459 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257462 2579 flags.go:64] FLAG: --logging-format="text" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257465 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257468 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257471 2579 flags.go:64] FLAG: --manifest-url="" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257473 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257478 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257482 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257486 2579 flags.go:64] FLAG: --max-pods="110" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257489 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257492 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257495 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257497 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257500 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257503 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257506 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 19:07:40.319871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257513 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257516 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257519 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257522 2579 flags.go:64] FLAG: --pod-cidr="" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257525 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257530 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257541 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257545 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257548 2579 flags.go:64] FLAG: --port="10250" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257551 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257553 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f45373829f1cf58b" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257558 2579 flags.go:64] FLAG: --qos-reserved="" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257561 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257564 2579 flags.go:64] FLAG: --register-node="true" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257567 2579 flags.go:64] FLAG: --register-with-taints="" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257571 2579 flags.go:64] FLAG: --registry-burst="10" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257574 2579 flags.go:64] FLAG: --registry-qps="5" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257577 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257580 2579 flags.go:64] FLAG: --reserved-memory="" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257584 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257586 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257590 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257592 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257595 2579 flags.go:64] FLAG: --runonce="false" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257599 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 19:07:40.320681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257602 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257605 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257608 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257611 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257613 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257616 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257619 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257622 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257624 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257627 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257630 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257633 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257636 2579 flags.go:64] FLAG: --system-cgroups="" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257639 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257644 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257647 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257664 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257671 2579 flags.go:64] FLAG: --tls-min-version="" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257675 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257678 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257681 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257684 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257687 2579 flags.go:64] FLAG: --v="2" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257691 2579 flags.go:64] FLAG: --version="false" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257696 2579 flags.go:64] FLAG: --vmodule="" Jul 10 19:07:40.437213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257699 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 19:07:40.334716 ip-10-0-132-221 systemd[1]: Started Kubernetes Kubelet. Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.257703 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257816 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257820 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257823 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257826 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257831 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257834 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257836 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257839 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257842 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257844 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257848 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257852 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257856 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257859 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257862 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257864 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257867 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:07:40.438509 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257870 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257873 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257875 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257878 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257881 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257883 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257888 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257891 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257893 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257896 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257898 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257901 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257903 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257906 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257909 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257911 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257914 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257916 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257919 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257923 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:07:40.439440 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257925 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257928 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257931 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257934 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257936 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257939 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257941 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257944 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257947 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257950 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257952 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257955 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257959 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257962 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257965 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257968 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257971 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257973 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257978 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:07:40.440675 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257981 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257984 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257986 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257989 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257991 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257994 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.257996 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258000 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258002 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258005 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258007 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258010 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258013 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258016 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258018 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258021 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258023 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258026 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258028 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258031 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:07:40.441560 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258034 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258036 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258039 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258041 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258044 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258046 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258049 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258051 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258054 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258056 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258059 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258062 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258065 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258067 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258070 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258072 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258075 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258078 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258080 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258083 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:07:40.442405 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258087 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258090 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.258093 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.258774 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.265208 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.265221 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.266711 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.266718 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266776 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266781 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266784 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266788 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266791 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266794 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266798 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:07:40.443274 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266801 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266804 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266807 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266809 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266812 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266815 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266817 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266820 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266822 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266825 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266829 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266833 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266836 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266839 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266842 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266845 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266848 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266851 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266853 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266856 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:07:40.444132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266858 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266863 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266867 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266869 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266872 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266875 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266877 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266880 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266882 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266885 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266887 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266890 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266893 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266896 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266898 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266901 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266903 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266906 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266909 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:07:40.445007 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266911 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266914 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266917 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266920 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266923 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266925 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266928 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266930 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266933 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266935 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266938 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266941 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266943 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266946 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266949 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266951 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266954 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266957 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266959 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:07:40.445905 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266961 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266964 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266966 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266970 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266973 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266976 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266979 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266981 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266984 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266987 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266989 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266992 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266995 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.266997 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267000 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267003 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267005 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267008 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267010 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:07:40.446710 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267013 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267015 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267018 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267021 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267024 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267026 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267029 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267031 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267034 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267036 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267039 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267042 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267044 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267047 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267050 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:07:40.447562 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.267055 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267160 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267165 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267168 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267171 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267174 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267176 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267179 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267182 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267185 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267188 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267193 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267196 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267198 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267201 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267204 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267206 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267209 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267211 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:07:40.448434 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267214 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267217 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267221 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267223 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267226 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267228 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267231 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267233 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267236 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267238 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267241 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267244 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267247 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267249 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267252 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267254 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267257 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267260 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267262 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267265 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:07:40.449298 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267267 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267270 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267273 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267275 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267280 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267283 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267285 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267287 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267290 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267293 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267295 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267299 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267302 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267305 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267309 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267312 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267315 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267318 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267321 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:07:40.450175 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267324 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267326 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267329 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267332 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267334 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267337 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267340 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267342 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267345 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267347 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267350 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267352 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267355 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267357 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267360 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267362 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267365 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267369 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267371 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267374 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:07:40.451048 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267376 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267379 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267382 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267384 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267387 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267390 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267392 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267396 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267399 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267401 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267404 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267406 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267409 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267411 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267414 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267417 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267419 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267422 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267426 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:07:40.451906 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267429 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267432 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:40.267435 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.267440 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.267525 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.269777 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.270926 2579 server.go:1019] "Starting client certificate rotation" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.271022 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.271084 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.273481 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.277937 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.296317 2579 log.go:25] "Validated CRI v1 runtime API" Jul 10 19:07:40.452777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.302938 2579 log.go:25] "Validated CRI v1 image API" Jul 10 19:07:40.453209 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.303787 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 19:07:40.453209 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.304551 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 19:07:40.453209 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.308337 2579 fs.go:135] Filesystem UUIDs: map[5deec391-f3c1-4094-9408-b6f9f2143288:/dev/nvme0n1p3 610604a8-28a0-44cc-a052-216071316a8c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 10 19:07:40.453209 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.308351 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.313008 2579 manager.go:216] Machine: {Timestamp:2026-07-10 19:07:40.311862718 +0000 UTC m=+0.350735725 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082950 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2627c44ed2bbee8c5676c77424d00b SystemUUID:ec2627c4-4ed2-bbee-8c56-76c77424d00b BootID:366a16e1-1a9f-4407-986d-622809db7d6f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7b:9b:61:2e:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7b:9b:61:2e:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:3e:f9:a9:a1:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.313105 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.313167 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.316505 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.316529 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.317296 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.317306 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.317313 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.318083 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.318982 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322189 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322297 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322323 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322332 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322343 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322351 2579 kubelet.go:398] "Adding apiserver pod source" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322360 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 19:07:40.458001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322734 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.322758 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.326326 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.327909 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.327959 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329256 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329275 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329285 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329293 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329304 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329313 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329324 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329332 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329343 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329351 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329363 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.329371 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.330146 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.330158 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.333689 2579 server.go:1268] "Started kubelet" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.334491 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.334562 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.335198 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.335273 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.335327 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.335493 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.335884 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j5n2v" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.338257 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.341969 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j5n2v" Jul 10 19:07:40.458877 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.351436 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-221.ec2.internal.18c103369255d935 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-221.ec2.internal,UID:ip-10-0-132-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-221.ec2.internal,},FirstTimestamp:2026-07-10 19:07:40.333644085 +0000 UTC m=+0.372517094,LastTimestamp:2026-07-10 19:07:40.333644085 +0000 UTC m=+0.372517094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-221.ec2.internal,}" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.521591 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.525287 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.525299 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.525572 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.525845 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.525848 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.525872 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.525996 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.526002 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.535367 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:07:40.571096 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.538510 2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-221.ec2.internal\" not found" node="ip-10-0-132-221.ec2.internal" Jul 10 19:07:40.688022 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.625639 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 10 19:07:40.808368 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.725851 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 10 19:07:40.811536 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.811283 2579 factory.go:55] Registering systemd factory Jul 10 19:07:40.811536 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.811319 2579 factory.go:223] Registration of the systemd container factory successfully Jul 10 19:07:40.811809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.811555 2579 factory.go:153] Registering CRI-O factory Jul 10 19:07:40.811809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.811571 2579 factory.go:223] Registration of the crio container factory successfully Jul 10 19:07:40.811809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.811625 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 19:07:40.811809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.811650 2579 factory.go:103] Registering Raw factory Jul 10 19:07:40.811809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.811714 2579 manager.go:1196] Started watching for new ooms in manager Jul 10 19:07:40.812583 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.812410 2579 manager.go:349] Starting recovery of all containers Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.825457 2579 manager.go:354] Recovery completed Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.825939 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-221.ec2.internal\" not found" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.829235 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.831294 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.831319 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.831329 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.831831 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.831835 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.831841 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.831859 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.832551 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.832582 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.832598 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.833228 2579 policy_none.go:49] "None policy: Start" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.833242 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.833252 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.834410 2579 policy_none.go:47] "Start" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.880292 2579 manager.go:337] "Starting Device Plugin manager" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.880456 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.880471 2579 server.go:87] "Starting device plugin registration server" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.880686 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.880717 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.880804 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.880886 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.880896 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.881275 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 19:07:40.968231 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.881321 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-221.ec2.internal\" not found" Jul 10 19:07:40.987939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.981357 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:07:40.987939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.982197 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:07:40.987939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.982222 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:07:40.987939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.982237 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:07:40.987939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.982264 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-221.ec2.internal" Jul 10 19:07:40.992808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:40.991937 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-221.ec2.internal" Jul 10 19:07:40.992808 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:40.991954 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-221.ec2.internal\": node \"ip-10-0-132-221.ec2.internal\" not found" Jul 10 19:07:41.121053 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.063385 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 19:07:41.121053 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.064596 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 19:07:41.121053 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.064631 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 19:07:41.121053 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.064649 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 19:07:41.121053 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:41.064679 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 19:07:41.121053 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.067213 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:07:41.252794 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:41.167039 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 19:07:41.252794 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.242404 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:07:41.271114 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.270207 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 19:07:41.271114 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.270319 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 19:07:41.271114 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.270372 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 19:07:41.271114 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.270372 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 19:07:41.387184 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.322738 2579 apiserver.go:52] "Watching apiserver" Jul 10 19:07:41.387184 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:41.367982 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 19:07:41.632220 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.525693 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 19:07:41.632220 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.525728 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 19:02:40 +0000 UTC" deadline="2028-03-07 03:16:14.039380989 +0000 UTC" Jul 10 19:07:41.632220 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.525758 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14528h8m32.513627048s" Jul 10 19:07:41.769050 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:41.769002 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 19:07:41.779092 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.779068 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:07:41.781424 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.781405 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 19:07:41.786830 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.786812 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 19:07:41.816776 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.816753 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zk4vj" Jul 10 19:07:41.826898 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:41.826879 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zk4vj" Jul 10 19:07:42.554697 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.554670 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:07:42.570056 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.570029 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal"] Jul 10 19:07:42.572418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.572395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.573630 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.573611 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-cj2bx","openshift-cluster-node-tuning-operator/tuned-zfpxd","openshift-image-registry/node-ca-kfwrf","openshift-network-diagnostics/network-check-target-zkfcv","openshift-network-operator/iptables-alerter-47tnf","kube-system/global-pull-secret-syncer-ssw5f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k","openshift-dns/node-resolver-sxcqz","openshift-multus/multus-4tsnc","openshift-multus/multus-additional-cni-plugins-4r25b","openshift-multus/network-metrics-daemon-xxz79","openshift-ovn-kubernetes/ovnkube-node-tm9q6"] Jul 10 19:07:42.573745 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.573719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.574871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.574859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.576130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.576113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.577222 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.577200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.578798 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.578469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:42.578798 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.578537 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:42.579768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.579752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.581146 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.580660 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jlzcw\"" Jul 10 19:07:42.581146 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.580812 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 19:07:42.581146 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.580949 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-76spq\"" Jul 10 19:07:42.581146 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.580989 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 19:07:42.581146 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.581059 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:07:42.581441 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.581263 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 19:07:42.582366 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.582195 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 19:07:42.582834 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.582813 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 19:07:42.582894 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.582879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.582943 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.582932 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:42.583301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.583284 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 19:07:42.584269 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.584251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.585396 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.585380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.585839 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.585823 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v8qcr\"" Jul 10 19:07:42.585895 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.585871 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:07:42.585895 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.585883 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 19:07:42.585980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.585963 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9rcm4\"" Jul 10 19:07:42.586742 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.586714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.586954 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.586935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-99pk4\"" Jul 10 19:07:42.587315 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.587300 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 19:07:42.588077 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.588063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.588235 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.588218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 19:07:42.588310 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.588272 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 19:07:42.589365 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.589351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:42.589431 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.589408 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:42.590585 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.590569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.594819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.594797 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 19:07:42.594819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.594810 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 19:07:42.594983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.594853 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 19:07:42.594983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.594863 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 19:07:42.594983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.594931 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 19:07:42.594983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.594953 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 19:07:42.595658 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.595641 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.596183 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.596164 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 19:07:42.596183 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.596182 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 19:07:42.596311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.596269 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 19:07:42.597490 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597470 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 19:07:42.597592 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597494 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 19:07:42.597592 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597475 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 19:07:42.597702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597579 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 19:07:42.597702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597596 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gzr28\"" Jul 10 19:07:42.597832 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597815 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 19:07:42.597878 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597856 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5b5qp\"" Jul 10 19:07:42.597960 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597937 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zhn8c\"" Jul 10 19:07:42.597960 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597953 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 19:07:42.598124 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.597946 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j546m\"" Jul 10 19:07:42.611001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.610978 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 19:07:42.611088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.611060 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal"] Jul 10 19:07:42.611228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.611216 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.615985 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.615970 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:07:42.619448 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.619432 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal"] Jul 10 19:07:42.619536 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.619528 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 19:07:42.626675 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.626659 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 19:07:42.637322 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-log-socket\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.637381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-cni-bin\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.637381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-run-ovn-kubernetes\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.637381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-lib-modules\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.637381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-device-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.637533 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-cni-binary-copy\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.637533 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.637533 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:42.637533 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-systemd\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.637533 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-kubelet-config\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.637533 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54573428-5115-497b-9056-2ebd4783c0a5-hosts-file\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-serviceca\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-cni-multus\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-hostroot\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-system-cni-dir\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-cnibin\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysconfig\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2tf9\" (UniqueName: \"kubernetes.io/projected/d35d1757-a739-4075-8e14-94cb02bdb9c6-kube-api-access-j2tf9\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6k2\" (UniqueName: \"kubernetes.io/projected/226ca90d-820d-4c55-b7b2-7f8099397399-kube-api-access-lx6k2\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:42.637790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-sys\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8prth\" (UniqueName: \"kubernetes.io/projected/37034b59-98ec-4f55-94b8-9d0dc6dffb20-kube-api-access-8prth\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-run-netns\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-ovn\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-node-log\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-cni-netd\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794cdb83c2901bbdd19f7c4480e67c66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"794cdb83c2901bbdd19f7c4480e67c66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.637975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-sys-fs\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-socket-dir-parent\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-netns\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-os-release\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-kubernetes\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-kubelet\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608-agent-certs\") pod \"konnectivity-agent-cj2bx\" (UID: \"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608\") " pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.638234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-systemd\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-system-cni-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-dbus\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a153daf-5790-42c6-9c33-98f40f104f91-ovn-node-metrics-cert\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-socket-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-os-release\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-tuned\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d1757-a739-4075-8e14-94cb02bdb9c6-host-slash\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-cni-bin\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-k8s-cni-cncf-io\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-daemon-config\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.638671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-run\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-var-lib-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-cnibin\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-conf-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysctl-d\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-ovnkube-script-lib\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37034b59-98ec-4f55-94b8-9d0dc6dffb20-tmp\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-registration-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqlv\" (UniqueName: \"kubernetes.io/projected/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-kube-api-access-msqlv\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grx7p\" (UniqueName: \"kubernetes.io/projected/159899a6-98e8-4ceb-9c8a-2b110bdf377e-kube-api-access-grx7p\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c868ebd70961df44ee76abf8462d8081-config\") pod \"kube-apiserver-proxy-ip-10-0-132-221.ec2.internal\" (UID: \"c868ebd70961df44ee76abf8462d8081\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mb4q\" (UniqueName: \"kubernetes.io/projected/54573428-5115-497b-9056-2ebd4783c0a5-kube-api-access-4mb4q\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-systemd-units\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.638980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk58t\" (UniqueName: \"kubernetes.io/projected/1a153daf-5790-42c6-9c33-98f40f104f91-kube-api-access-rk58t\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-cni-binary-copy\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.639119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-modprobe-d\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54573428-5115-497b-9056-2ebd4783c0a5-tmp-dir\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-slash\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-cni-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-var-lib-kubelet\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxw5\" (UniqueName: \"kubernetes.io/projected/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-kube-api-access-6cxw5\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-ovnkube-config\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-env-overrides\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/794cdb83c2901bbdd19f7c4480e67c66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"794cdb83c2901bbdd19f7c4480e67c66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-kubelet\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysctl-conf\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-host\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d35d1757-a739-4075-8e14-94cb02bdb9c6-iptables-alerter-script\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-etc-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608-konnectivity-ca\") pod \"konnectivity-agent-cj2bx\" (UID: \"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608\") " pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:42.639558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-multus-certs\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.640267 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-etc-kubernetes\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.640267 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4kc\" (UniqueName: \"kubernetes.io/projected/1309dd30-f208-401e-a5e0-b33b8a018b90-kube-api-access-9v4kc\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.640267 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.639330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-host\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.740668 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk58t\" (UniqueName: \"kubernetes.io/projected/1a153daf-5790-42c6-9c33-98f40f104f91-kube-api-access-rk58t\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.740767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-cni-binary-copy\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.740767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-modprobe-d\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.740767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54573428-5115-497b-9056-2ebd4783c0a5-tmp-dir\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.740767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-slash\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.740767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-cni-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.741027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-var-lib-kubelet\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.741027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-slash\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.741027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxw5\" (UniqueName: \"kubernetes.io/projected/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-kube-api-access-6cxw5\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.741027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-modprobe-d\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.741027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-var-lib-kubelet\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.741027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.740928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-cni-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.741027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-ovnkube-config\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-env-overrides\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/794cdb83c2901bbdd19f7c4480e67c66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"794cdb83c2901bbdd19f7c4480e67c66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741093 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54573428-5115-497b-9056-2ebd4783c0a5-tmp-dir\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/794cdb83c2901bbdd19f7c4480e67c66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"794cdb83c2901bbdd19f7c4480e67c66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-kubelet\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysctl-conf\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-host\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-host\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-kubelet\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d35d1757-a739-4075-8e14-94cb02bdb9c6-iptables-alerter-script\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysctl-conf\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-etc-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608-konnectivity-ca\") pod \"konnectivity-agent-cj2bx\" (UID: \"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608\") " pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-multus-certs\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741415 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-cni-binary-copy\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.741442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-etc-kubernetes\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4kc\" (UniqueName: \"kubernetes.io/projected/1309dd30-f208-401e-a5e0-b33b8a018b90-kube-api-access-9v4kc\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-host\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-multus-certs\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-log-socket\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-cni-bin\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-run-ovn-kubernetes\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-lib-modules\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-device-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-cni-binary-copy\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-env-overrides\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-cni-bin\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-systemd\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-ovnkube-config\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-kubelet-config\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.742283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-host\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-etc-kubernetes\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-device-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54573428-5115-497b-9056-2ebd4783c0a5-hosts-file\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-run-ovn-kubernetes\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741849 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-kubelet-config\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54573428-5115-497b-9056-2ebd4783c0a5-hosts-file\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-log-socket\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d35d1757-a739-4075-8e14-94cb02bdb9c6-iptables-alerter-script\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-serviceca\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608-konnectivity-ca\") pod \"konnectivity-agent-cj2bx\" (UID: \"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608\") " pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.741937 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-systemd\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-cni-multus\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-etc-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.741922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-lib-modules\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.742995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-hostroot\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-var-lib-cni-multus\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-hostroot\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.742070 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs podName:226ca90d-820d-4c55-b7b2-7f8099397399 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:43.24204939 +0000 UTC m=+3.280922409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs") pod "network-metrics-daemon-xxz79" (UID: "226ca90d-820d-4c55-b7b2-7f8099397399") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-system-cni-dir\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-cnibin\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysconfig\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-system-cni-dir\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-cnibin\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2tf9\" (UniqueName: \"kubernetes.io/projected/d35d1757-a739-4075-8e14-94cb02bdb9c6-kube-api-access-j2tf9\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysconfig\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6k2\" (UniqueName: \"kubernetes.io/projected/226ca90d-820d-4c55-b7b2-7f8099397399-kube-api-access-lx6k2\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-sys\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8prth\" (UniqueName: \"kubernetes.io/projected/37034b59-98ec-4f55-94b8-9d0dc6dffb20-kube-api-access-8prth\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-run-netns\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-ovn\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742345 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-sys\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.744004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-node-log\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-serviceca\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-node-log\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-cni-netd\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-run-netns\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-ovn\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794cdb83c2901bbdd19f7c4480e67c66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"794cdb83c2901bbdd19f7c4480e67c66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-cni-netd\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-cni-binary-copy\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742509 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-sys-fs\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794cdb83c2901bbdd19f7c4480e67c66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal\" (UID: \"794cdb83c2901bbdd19f7c4480e67c66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-socket-dir-parent\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-sys-fs\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-netns\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.744811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-os-release\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-socket-dir-parent\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-kubernetes\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-netns\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-kubelet\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-kubernetes\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-os-release\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-kubelet\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608-agent-certs\") pod \"konnectivity-agent-cj2bx\" (UID: \"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608\") " pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-systemd\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-system-cni-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-dbus\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.742886 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-system-cni-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745361 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.742941 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret podName:d1acace1-4b9b-402d-a6ab-f8d8f3a401cc nodeName:}" failed. No retries permitted until 2026-07-10 19:07:43.242928124 +0000 UTC m=+3.281801122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret") pod "global-pull-secret-syncer-ssw5f" (UID: "d1acace1-4b9b-402d-a6ab-f8d8f3a401cc") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a153daf-5790-42c6-9c33-98f40f104f91-ovn-node-metrics-cert\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-run-systemd\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-dbus\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-socket-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.742999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-os-release\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-tuned\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d1757-a739-4075-8e14-94cb02bdb9c6-host-slash\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-socket-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d1757-a739-4075-8e14-94cb02bdb9c6-host-slash\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-os-release\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-cni-bin\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743113 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-k8s-cni-cncf-io\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-daemon-config\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-cni-bin\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-host-run-k8s-cni-cncf-io\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.745897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-run\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-var-lib-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-cnibin\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-conf-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1309dd30-f208-401e-a5e0-b33b8a018b90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysctl-d\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-var-lib-openvswitch\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-ovnkube-script-lib\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-run\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37034b59-98ec-4f55-94b8-9d0dc6dffb20-tmp\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-registration-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-cnibin\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqlv\" (UniqueName: \"kubernetes.io/projected/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-kube-api-access-msqlv\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grx7p\" (UniqueName: \"kubernetes.io/projected/159899a6-98e8-4ceb-9c8a-2b110bdf377e-kube-api-access-grx7p\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.746645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-sysctl-d\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c868ebd70961df44ee76abf8462d8081-config\") pod \"kube-apiserver-proxy-ip-10-0-132-221.ec2.internal\" (UID: \"c868ebd70961df44ee76abf8462d8081\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mb4q\" (UniqueName: \"kubernetes.io/projected/54573428-5115-497b-9056-2ebd4783c0a5-kube-api-access-4mb4q\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-systemd-units\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a153daf-5790-42c6-9c33-98f40f104f91-systemd-units\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-daemon-config\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743463 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-multus-conf-dir\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/159899a6-98e8-4ceb-9c8a-2b110bdf377e-registration-dir\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.743778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c868ebd70961df44ee76abf8462d8081-config\") pod \"kube-apiserver-proxy-ip-10-0-132-221.ec2.internal\" (UID: \"c868ebd70961df44ee76abf8462d8081\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.744048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a153daf-5790-42c6-9c33-98f40f104f91-ovnkube-script-lib\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.744585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1309dd30-f208-401e-a5e0-b33b8a018b90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.746546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37034b59-98ec-4f55-94b8-9d0dc6dffb20-etc-tuned\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.746595 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37034b59-98ec-4f55-94b8-9d0dc6dffb20-tmp\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.746785 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a153daf-5790-42c6-9c33-98f40f104f91-ovn-node-metrics-cert\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.747157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.746904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608-agent-certs\") pod \"konnectivity-agent-cj2bx\" (UID: \"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608\") " pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.758305 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.758287 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:07:42.758305 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.758306 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:07:42.758305 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.758315 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kl5wj for pod openshift-network-diagnostics/network-check-target-zkfcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:42.758547 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:42.758370 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj podName:8f760c7e-ca57-4789-a78b-327fdec3f6b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:43.258352926 +0000 UTC m=+3.297225954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kl5wj" (UniqueName: "kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj") pod "network-check-target-zkfcv" (UID: "8f760c7e-ca57-4789-a78b-327fdec3f6b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:42.760831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.760401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2tf9\" (UniqueName: \"kubernetes.io/projected/d35d1757-a739-4075-8e14-94cb02bdb9c6-kube-api-access-j2tf9\") pod \"iptables-alerter-47tnf\" (UID: \"d35d1757-a739-4075-8e14-94cb02bdb9c6\") " pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.760937 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.760898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4kc\" (UniqueName: \"kubernetes.io/projected/1309dd30-f208-401e-a5e0-b33b8a018b90-kube-api-access-9v4kc\") pod \"multus-additional-cni-plugins-4r25b\" (UID: \"1309dd30-f208-401e-a5e0-b33b8a018b90\") " pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:42.760937 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.760920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxw5\" (UniqueName: \"kubernetes.io/projected/c5a5c819-e84a-4d88-8c7a-8be2fa966e33-kube-api-access-6cxw5\") pod \"node-ca-kfwrf\" (UID: \"c5a5c819-e84a-4d88-8c7a-8be2fa966e33\") " pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.761059 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.761009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk58t\" (UniqueName: \"kubernetes.io/projected/1a153daf-5790-42c6-9c33-98f40f104f91-kube-api-access-rk58t\") pod \"ovnkube-node-tm9q6\" (UID: \"1a153daf-5790-42c6-9c33-98f40f104f91\") " pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:42.761432 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.761413 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mb4q\" (UniqueName: \"kubernetes.io/projected/54573428-5115-497b-9056-2ebd4783c0a5-kube-api-access-4mb4q\") pod \"node-resolver-sxcqz\" (UID: \"54573428-5115-497b-9056-2ebd4783c0a5\") " pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:42.762214 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.762192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqlv\" (UniqueName: \"kubernetes.io/projected/6a34835a-6cb8-4fb1-a5c8-445574ab2fa1-kube-api-access-msqlv\") pod \"multus-4tsnc\" (UID: \"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1\") " pod="openshift-multus/multus-4tsnc" Jul 10 19:07:42.762449 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.762432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grx7p\" (UniqueName: \"kubernetes.io/projected/159899a6-98e8-4ceb-9c8a-2b110bdf377e-kube-api-access-grx7p\") pod \"aws-ebs-csi-driver-node-zr74k\" (UID: \"159899a6-98e8-4ceb-9c8a-2b110bdf377e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:42.762926 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.762910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8prth\" (UniqueName: \"kubernetes.io/projected/37034b59-98ec-4f55-94b8-9d0dc6dffb20-kube-api-access-8prth\") pod \"tuned-zfpxd\" (UID: \"37034b59-98ec-4f55-94b8-9d0dc6dffb20\") " pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.763535 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.763518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6k2\" (UniqueName: \"kubernetes.io/projected/226ca90d-820d-4c55-b7b2-7f8099397399-kube-api-access-lx6k2\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:42.828786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.828701 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 19:02:42 +0000 UTC" deadline="2028-01-06 12:00:19.234181765 +0000 UTC" Jul 10 19:07:42.828786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.828738 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13072h52m36.405445993s" Jul 10 19:07:42.914970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.914937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.923552 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.923528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" Jul 10 19:07:42.930829 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.930808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:07:42.949272 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.949250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" Jul 10 19:07:42.962119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.962099 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfwrf" Jul 10 19:07:42.975989 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.975965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-47tnf" Jul 10 19:07:42.995835 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:42.995804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" Jul 10 19:07:43.014055 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.014031 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sxcqz" Jul 10 19:07:43.028798 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.028782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4tsnc" Jul 10 19:07:43.046803 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.046784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4r25b" Jul 10 19:07:43.056194 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.056173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:07:43.247850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.247821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:43.248019 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.247877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:43.248019 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.247985 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:43.248019 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.248013 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:43.248127 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.248053 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs podName:226ca90d-820d-4c55-b7b2-7f8099397399 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:44.24803009 +0000 UTC m=+4.286903102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs") pod "network-metrics-daemon-xxz79" (UID: "226ca90d-820d-4c55-b7b2-7f8099397399") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:43.248127 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.248070 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret podName:d1acace1-4b9b-402d-a6ab-f8d8f3a401cc nodeName:}" failed. No retries permitted until 2026-07-10 19:07:44.248063216 +0000 UTC m=+4.286936212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret") pod "global-pull-secret-syncer-ssw5f" (UID: "d1acace1-4b9b-402d-a6ab-f8d8f3a401cc") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:43.348203 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.348173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:43.348366 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.348347 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:07:43.348439 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.348377 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:07:43.348439 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.348391 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kl5wj for pod openshift-network-diagnostics/network-check-target-zkfcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:43.348549 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:43.348475 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj podName:8f760c7e-ca57-4789-a78b-327fdec3f6b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:44.348438885 +0000 UTC m=+4.387311899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl5wj" (UniqueName: "kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj") pod "network-check-target-zkfcv" (UID: "8f760c7e-ca57-4789-a78b-327fdec3f6b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:43.482767 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.482716 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35d1757_a739_4075_8e14_94cb02bdb9c6.slice/crio-58785f15beb95a4df11b7cc3aa74569987da0b2246a68b2a7f174a73b6f58421 WatchSource:0}: Error finding container 58785f15beb95a4df11b7cc3aa74569987da0b2246a68b2a7f174a73b6f58421: Status 404 returned error can't find the container with id 58785f15beb95a4df11b7cc3aa74569987da0b2246a68b2a7f174a73b6f58421 Jul 10 19:07:43.484212 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.484189 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37034b59_98ec_4f55_94b8_9d0dc6dffb20.slice/crio-0104ae7f22182a4409ab138d5b38335a510e78fae99e83a32fd5885be53c7117 WatchSource:0}: Error finding container 0104ae7f22182a4409ab138d5b38335a510e78fae99e83a32fd5885be53c7117: Status 404 returned error can't find the container with id 0104ae7f22182a4409ab138d5b38335a510e78fae99e83a32fd5885be53c7117 Jul 10 19:07:43.484860 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.484826 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a153daf_5790_42c6_9c33_98f40f104f91.slice/crio-fa625dfefd3bad8b8b56de218d760a0188d26c269b07c2e59b61f71d292c35d8 WatchSource:0}: Error finding container fa625dfefd3bad8b8b56de218d760a0188d26c269b07c2e59b61f71d292c35d8: Status 404 returned error can't find the container with id fa625dfefd3bad8b8b56de218d760a0188d26c269b07c2e59b61f71d292c35d8 Jul 10 19:07:43.485747 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.485551 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54573428_5115_497b_9056_2ebd4783c0a5.slice/crio-20affb5741094bdd42516bdfea7e42c74c4ab2b7e61e68e15dd175822227dfb2 WatchSource:0}: Error finding container 20affb5741094bdd42516bdfea7e42c74c4ab2b7e61e68e15dd175822227dfb2: Status 404 returned error can't find the container with id 20affb5741094bdd42516bdfea7e42c74c4ab2b7e61e68e15dd175822227dfb2 Jul 10 19:07:43.488815 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.488798 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:07:43.518836 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.518812 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a5c819_e84a_4d88_8c7a_8be2fa966e33.slice/crio-7fbbc18f8bbb46ee69e64fbf03f57b1c53b2e732ebe6f194995c8b16b67987d5 WatchSource:0}: Error finding container 7fbbc18f8bbb46ee69e64fbf03f57b1c53b2e732ebe6f194995c8b16b67987d5: Status 404 returned error can't find the container with id 7fbbc18f8bbb46ee69e64fbf03f57b1c53b2e732ebe6f194995c8b16b67987d5 Jul 10 19:07:43.535528 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.535502 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1309dd30_f208_401e_a5e0_b33b8a018b90.slice/crio-5a116fe1d3d7366a57d485c5c5375bedb01dabe8fe79cbab7b02ea10c724dda1 WatchSource:0}: Error finding container 5a116fe1d3d7366a57d485c5c5375bedb01dabe8fe79cbab7b02ea10c724dda1: Status 404 returned error can't find the container with id 5a116fe1d3d7366a57d485c5c5375bedb01dabe8fe79cbab7b02ea10c724dda1 Jul 10 19:07:43.535948 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.535911 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a34835a_6cb8_4fb1_a5c8_445574ab2fa1.slice/crio-247f3aa8a37673b4973a0d5cebbf13fdacefc22ace8495c2f1b57eaea06dfed0 WatchSource:0}: Error finding container 247f3aa8a37673b4973a0d5cebbf13fdacefc22ace8495c2f1b57eaea06dfed0: Status 404 returned error can't find the container with id 247f3aa8a37673b4973a0d5cebbf13fdacefc22ace8495c2f1b57eaea06dfed0 Jul 10 19:07:43.554707 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.554686 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:07:43.557328 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.557306 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159899a6_98e8_4ceb_9c8a_2b110bdf377e.slice/crio-ba28f3169bd9bee7c228565e12e0b73dd4ac2288ad3fa64924efcb868c358096 WatchSource:0}: Error finding container ba28f3169bd9bee7c228565e12e0b73dd4ac2288ad3fa64924efcb868c358096: Status 404 returned error can't find the container with id ba28f3169bd9bee7c228565e12e0b73dd4ac2288ad3fa64924efcb868c358096 Jul 10 19:07:43.615453 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.615424 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794cdb83c2901bbdd19f7c4480e67c66.slice/crio-56275951842ddeae07ec60f1636bf76c86fc322d5943efea4e50357f87823a2a WatchSource:0}: Error finding container 56275951842ddeae07ec60f1636bf76c86fc322d5943efea4e50357f87823a2a: Status 404 returned error can't find the container with id 56275951842ddeae07ec60f1636bf76c86fc322d5943efea4e50357f87823a2a Jul 10 19:07:43.669136 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.669109 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f3ee5e_5fae_4ce8_b74f_c0d34d1b8608.slice/crio-04cf6e5d0fc71a89bef0273250f3304c632524ab008812177592ce1c88a4faa4 WatchSource:0}: Error finding container 04cf6e5d0fc71a89bef0273250f3304c632524ab008812177592ce1c88a4faa4: Status 404 returned error can't find the container with id 04cf6e5d0fc71a89bef0273250f3304c632524ab008812177592ce1c88a4faa4 Jul 10 19:07:43.767011 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:07:43.766950 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc868ebd70961df44ee76abf8462d8081.slice/crio-992dd8b62a638cf56ff603f899bb2bb494e7de62ee1d17262c5545a725d6ab82 WatchSource:0}: Error finding container 992dd8b62a638cf56ff603f899bb2bb494e7de62ee1d17262c5545a725d6ab82: Status 404 returned error can't find the container with id 992dd8b62a638cf56ff603f899bb2bb494e7de62ee1d17262c5545a725d6ab82 Jul 10 19:07:43.774344 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.774301 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sxcqz" event={"ID":"54573428-5115-497b-9056-2ebd4783c0a5","Type":"ContainerStarted","Data":"20affb5741094bdd42516bdfea7e42c74c4ab2b7e61e68e15dd175822227dfb2"} Jul 10 19:07:43.775442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.775419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"fa625dfefd3bad8b8b56de218d760a0188d26c269b07c2e59b61f71d292c35d8"} Jul 10 19:07:43.776526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.776505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-47tnf" event={"ID":"d35d1757-a739-4075-8e14-94cb02bdb9c6","Type":"ContainerStarted","Data":"58785f15beb95a4df11b7cc3aa74569987da0b2246a68b2a7f174a73b6f58421"} Jul 10 19:07:43.777606 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.777587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" event={"ID":"37034b59-98ec-4f55-94b8-9d0dc6dffb20","Type":"ContainerStarted","Data":"0104ae7f22182a4409ab138d5b38335a510e78fae99e83a32fd5885be53c7117"} Jul 10 19:07:43.778648 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.778627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" event={"ID":"794cdb83c2901bbdd19f7c4480e67c66","Type":"ContainerStarted","Data":"56275951842ddeae07ec60f1636bf76c86fc322d5943efea4e50357f87823a2a"} Jul 10 19:07:43.779632 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.779612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" event={"ID":"159899a6-98e8-4ceb-9c8a-2b110bdf377e","Type":"ContainerStarted","Data":"ba28f3169bd9bee7c228565e12e0b73dd4ac2288ad3fa64924efcb868c358096"} Jul 10 19:07:43.780571 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.780546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4tsnc" event={"ID":"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1","Type":"ContainerStarted","Data":"247f3aa8a37673b4973a0d5cebbf13fdacefc22ace8495c2f1b57eaea06dfed0"} Jul 10 19:07:43.781587 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.781565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerStarted","Data":"5a116fe1d3d7366a57d485c5c5375bedb01dabe8fe79cbab7b02ea10c724dda1"} Jul 10 19:07:43.782615 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.782596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" event={"ID":"c868ebd70961df44ee76abf8462d8081","Type":"ContainerStarted","Data":"992dd8b62a638cf56ff603f899bb2bb494e7de62ee1d17262c5545a725d6ab82"} Jul 10 19:07:43.783613 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.783595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cj2bx" event={"ID":"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608","Type":"ContainerStarted","Data":"04cf6e5d0fc71a89bef0273250f3304c632524ab008812177592ce1c88a4faa4"} Jul 10 19:07:43.784673 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.784638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfwrf" event={"ID":"c5a5c819-e84a-4d88-8c7a-8be2fa966e33","Type":"ContainerStarted","Data":"7fbbc18f8bbb46ee69e64fbf03f57b1c53b2e732ebe6f194995c8b16b67987d5"} Jul 10 19:07:43.829586 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.829542 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 19:02:42 +0000 UTC" deadline="2028-04-03 06:02:39.661052166 +0000 UTC" Jul 10 19:07:43.829586 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:43.829583 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15178h54m55.831472446s" Jul 10 19:07:44.065539 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:44.065463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:44.065704 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.065598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:44.066049 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:44.066032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:44.066154 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.066132 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:44.066221 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:44.066207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:44.066355 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.066335 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:44.256766 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:44.256733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:44.256950 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:44.256791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:44.256950 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.256877 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:44.256950 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.256916 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:44.257125 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.256952 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs podName:226ca90d-820d-4c55-b7b2-7f8099397399 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:46.25692843 +0000 UTC m=+6.295801427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs") pod "network-metrics-daemon-xxz79" (UID: "226ca90d-820d-4c55-b7b2-7f8099397399") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:44.257125 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.256976 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret podName:d1acace1-4b9b-402d-a6ab-f8d8f3a401cc nodeName:}" failed. No retries permitted until 2026-07-10 19:07:46.256965348 +0000 UTC m=+6.295838347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret") pod "global-pull-secret-syncer-ssw5f" (UID: "d1acace1-4b9b-402d-a6ab-f8d8f3a401cc") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:44.357503 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:44.357418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:44.357707 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.357653 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:07:44.357707 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.357678 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:07:44.357856 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.357712 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kl5wj for pod openshift-network-diagnostics/network-check-target-zkfcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:44.357856 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:44.357773 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj podName:8f760c7e-ca57-4789-a78b-327fdec3f6b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:46.357756616 +0000 UTC m=+6.396629613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl5wj" (UniqueName: "kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj") pod "network-check-target-zkfcv" (UID: "8f760c7e-ca57-4789-a78b-327fdec3f6b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:46.068010 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:46.067214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:46.068010 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.067342 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:46.068010 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:46.067726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:46.068010 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.067817 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:46.068010 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:46.067894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:46.068010 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.067976 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:46.275061 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:46.275021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:46.275228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:46.275107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:46.275305 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.275226 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:46.275357 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.275308 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs podName:226ca90d-820d-4c55-b7b2-7f8099397399 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:50.275266524 +0000 UTC m=+10.314139520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs") pod "network-metrics-daemon-xxz79" (UID: "226ca90d-820d-4c55-b7b2-7f8099397399") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:46.275809 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.275710 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:46.275809 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.275765 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret podName:d1acace1-4b9b-402d-a6ab-f8d8f3a401cc nodeName:}" failed. No retries permitted until 2026-07-10 19:07:50.275747952 +0000 UTC m=+10.314620948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret") pod "global-pull-secret-syncer-ssw5f" (UID: "d1acace1-4b9b-402d-a6ab-f8d8f3a401cc") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:46.375727 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:46.375691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:46.375909 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.375890 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:07:46.375981 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.375916 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:07:46.375981 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.375930 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kl5wj for pod openshift-network-diagnostics/network-check-target-zkfcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:46.376092 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:46.375983 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj podName:8f760c7e-ca57-4789-a78b-327fdec3f6b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:50.375965752 +0000 UTC m=+10.414838766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl5wj" (UniqueName: "kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj") pod "network-check-target-zkfcv" (UID: "8f760c7e-ca57-4789-a78b-327fdec3f6b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:48.064961 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:48.064928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:48.065413 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:48.065067 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:48.065476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:48.065459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:48.065777 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:48.065578 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:48.065777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:48.065644 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:48.065777 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:48.065734 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:50.065850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:50.065819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:50.066305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:50.065931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:50.066305 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.065944 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:50.066305 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.066045 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:50.066305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:50.066097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:50.066305 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.066177 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:50.305814 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:50.305783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:50.305993 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:50.305847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:50.305993 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.305956 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:50.306110 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.306027 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs podName:226ca90d-820d-4c55-b7b2-7f8099397399 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:58.306006313 +0000 UTC m=+18.344879323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs") pod "network-metrics-daemon-xxz79" (UID: "226ca90d-820d-4c55-b7b2-7f8099397399") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:50.306110 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.305956 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:50.306110 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.306082 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret podName:d1acace1-4b9b-402d-a6ab-f8d8f3a401cc nodeName:}" failed. No retries permitted until 2026-07-10 19:07:58.306068741 +0000 UTC m=+18.344941740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret") pod "global-pull-secret-syncer-ssw5f" (UID: "d1acace1-4b9b-402d-a6ab-f8d8f3a401cc") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:50.406154 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:50.406122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:50.406338 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.406306 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:07:50.406338 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.406329 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:07:50.406462 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.406342 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kl5wj for pod openshift-network-diagnostics/network-check-target-zkfcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:50.406462 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:50.406402 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj podName:8f760c7e-ca57-4789-a78b-327fdec3f6b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:07:58.406383298 +0000 UTC m=+18.445256308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl5wj" (UniqueName: "kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj") pod "network-check-target-zkfcv" (UID: "8f760c7e-ca57-4789-a78b-327fdec3f6b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:52.065948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:52.065429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:52.065948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:52.065463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:52.065948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:52.065464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:52.065948 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:52.065563 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:52.065948 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:52.065617 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:52.065948 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:52.065692 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:54.065319 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:54.065277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:54.065319 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:54.065306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:54.065810 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:54.065308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:54.065810 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:54.065401 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:54.065810 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:54.065482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:54.065810 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:54.065563 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:56.065738 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:56.065706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:56.065738 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:56.065739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:56.066223 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:56.065813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:56.066223 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:56.065826 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:56.066223 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:56.065897 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:56.066223 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:56.065999 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:58.065907 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:58.065713 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:58.066365 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:58.065725 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:58.066365 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.066014 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:07:58.066365 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.066056 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:07:58.066365 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:58.065762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:58.066365 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.066261 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:07:58.361899 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:58.361868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:07:58.362083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:58.361927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:07:58.362083 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.362038 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:58.362083 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.362049 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:58.362248 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.362113 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs podName:226ca90d-820d-4c55-b7b2-7f8099397399 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:14.362091523 +0000 UTC m=+34.400964535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs") pod "network-metrics-daemon-xxz79" (UID: "226ca90d-820d-4c55-b7b2-7f8099397399") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:07:58.362248 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.362136 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret podName:d1acace1-4b9b-402d-a6ab-f8d8f3a401cc nodeName:}" failed. No retries permitted until 2026-07-10 19:08:14.362124488 +0000 UTC m=+34.400997490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret") pod "global-pull-secret-syncer-ssw5f" (UID: "d1acace1-4b9b-402d-a6ab-f8d8f3a401cc") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:07:58.462884 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:07:58.462842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:07:58.463045 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.463023 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:07:58.463110 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.463046 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:07:58.463110 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.463058 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kl5wj for pod openshift-network-diagnostics/network-check-target-zkfcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:07:58.463208 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:07:58.463116 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj podName:8f760c7e-ca57-4789-a78b-327fdec3f6b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:14.46309621 +0000 UTC m=+34.501969211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl5wj" (UniqueName: "kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj") pod "network-check-target-zkfcv" (UID: "8f760c7e-ca57-4789-a78b-327fdec3f6b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:08:00.064816 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:00.064785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:00.064816 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:00.064811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:00.065280 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:00.064819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:00.065280 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:00.064896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:00.065280 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:00.065062 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:00.065280 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:00.065164 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:02.065360 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:02.065329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:02.065792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:02.065362 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:02.065792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:02.065332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:02.065792 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:02.065475 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:02.065792 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:02.065595 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:02.065792 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:02.065715 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:04.065860 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.065697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:04.066228 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:04.065969 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:04.066228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.065772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:04.066228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.065728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:04.066228 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:04.066059 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:04.066634 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:04.066599 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:04.858863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.858839 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:08:04.859165 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.859144 2579 generic.go:356] "Generic (PLEG): container finished" podID="1a153daf-5790-42c6-9c33-98f40f104f91" containerID="a364d1186336d6b087e42e3cb1d3888f19035f1db140a7d43dd265529ff43082" exitCode=1 Jul 10 19:08:04.859252 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.859216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"8d379b41d069f7c9a3890b20ff717e0f606bc5c8de338da635609224c9ab8884"} Jul 10 19:08:04.859304 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.859276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"beec54e8aaeb6a20e9c2ee109b7e287d8617032f218e6d8f87e760b37577ef61"} Jul 10 19:08:04.859356 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.859315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"14e820ada8141ef55cb5937ca1d819371aad6c2be7a4f1062d884eb9782e5f89"} Jul 10 19:08:04.859397 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.859372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"f96f251f6fc8ef91854abbd3b21e2aa3f85ee644fde8f142ae7ebddb79affa5e"} Jul 10 19:08:04.859445 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.859410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerDied","Data":"a364d1186336d6b087e42e3cb1d3888f19035f1db140a7d43dd265529ff43082"} Jul 10 19:08:04.859500 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.859451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"5a36eff0595c518998c687dca30e469137d78ecd1c8dc3e7f3bf2c32d2aa6884"} Jul 10 19:08:04.860568 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.860545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-47tnf" event={"ID":"d35d1757-a739-4075-8e14-94cb02bdb9c6","Type":"ContainerStarted","Data":"9628cb473620b98a13cb9a66c1629c312fed177c58b6beb2f7fa1c497fb76eee"} Jul 10 19:08:04.861860 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.861840 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" event={"ID":"37034b59-98ec-4f55-94b8-9d0dc6dffb20","Type":"ContainerStarted","Data":"5f38eca3e11fbed11fb9512bf0c2aebc61f5ef1b8ff85298d78d3894407cc73a"} Jul 10 19:08:04.863218 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.863196 2579 generic.go:356] "Generic (PLEG): container finished" podID="794cdb83c2901bbdd19f7c4480e67c66" containerID="8b601f0b75ee3af9920b1a54e608061b639bc6a2b574d3fd6f07bab732ac9c66" exitCode=0 Jul 10 19:08:04.863318 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.863260 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" event={"ID":"794cdb83c2901bbdd19f7c4480e67c66","Type":"ContainerDied","Data":"8b601f0b75ee3af9920b1a54e608061b639bc6a2b574d3fd6f07bab732ac9c66"} Jul 10 19:08:04.864745 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.864719 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" event={"ID":"159899a6-98e8-4ceb-9c8a-2b110bdf377e","Type":"ContainerStarted","Data":"fe19c7b7794138c5180d324bf8ef8d4dbbdabe16921e41703be080a0becc980e"} Jul 10 19:08:04.866135 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.866112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4tsnc" event={"ID":"6a34835a-6cb8-4fb1-a5c8-445574ab2fa1","Type":"ContainerStarted","Data":"2ffee41234eebe5944869f589858e85f4fc5d5a6f00728b10758461c603abda4"} Jul 10 19:08:04.867450 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.867430 2579 generic.go:356] "Generic (PLEG): container finished" podID="1309dd30-f208-401e-a5e0-b33b8a018b90" containerID="34cb20a36631784b03633e6aa58f7e41627f4a69e5f1f48c184d634c2fb88143" exitCode=0 Jul 10 19:08:04.867549 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.867493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerDied","Data":"34cb20a36631784b03633e6aa58f7e41627f4a69e5f1f48c184d634c2fb88143"} Jul 10 19:08:04.868792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.868769 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" event={"ID":"c868ebd70961df44ee76abf8462d8081","Type":"ContainerStarted","Data":"22fb85f683b2d09c9e441957edea068b28edfe63daa748eb646c660527e74d2e"} Jul 10 19:08:04.870122 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.870100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cj2bx" event={"ID":"86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608","Type":"ContainerStarted","Data":"ddf7b0b2f914be6458e08ac393e3f11d08847654ac5fb93d349399ba868a3855"} Jul 10 19:08:04.871561 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.871544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfwrf" event={"ID":"c5a5c819-e84a-4d88-8c7a-8be2fa966e33","Type":"ContainerStarted","Data":"1dfdb1866d159f2a96e24f7df0a1277e368cec5ad5bea28cd7f2d86b5d7864a4"} Jul 10 19:08:04.872996 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.872977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sxcqz" event={"ID":"54573428-5115-497b-9056-2ebd4783c0a5","Type":"ContainerStarted","Data":"c8ae987b2e308ad1d7211648f489503b5fbcb8ee8897ff583e39d3266242cd7b"} Jul 10 19:08:04.878945 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.877459 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-47tnf" podStartSLOduration=3.843332419 podStartE2EDuration="23.877440619s" totalImagesPullingTime="20.0341082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:04.87649913 +0000 UTC m=+24.915372148" watchObservedRunningTime="2026-07-10 19:08:04.877440619 +0000 UTC m=+24.916313637" Jul 10 19:08:04.892565 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.892518 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kfwrf" podStartSLOduration=8.49373342 podStartE2EDuration="23.89250276s" totalImagesPullingTime="15.39876934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:04.891719057 +0000 UTC m=+24.930592070" watchObservedRunningTime="2026-07-10 19:08:04.89250276 +0000 UTC m=+24.931375780" Jul 10 19:08:04.909233 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.909200 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4tsnc" podStartSLOduration=3.499426666 podStartE2EDuration="23.90918901s" totalImagesPullingTime="20.409762344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:04.908992219 +0000 UTC m=+24.947865236" watchObservedRunningTime="2026-07-10 19:08:04.90918901 +0000 UTC m=+24.948062027" Jul 10 19:08:04.924638 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.924605 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cj2bx" podStartSLOduration=4.089838359 podStartE2EDuration="23.924597109s" totalImagesPullingTime="19.83475875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:04.9241607 +0000 UTC m=+24.963033718" watchObservedRunningTime="2026-07-10 19:08:04.924597109 +0000 UTC m=+24.963470126" Jul 10 19:08:04.942195 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.942152 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zfpxd" podStartSLOduration=3.905992054 podStartE2EDuration="23.942142529s" totalImagesPullingTime="20.036150475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:04.941397931 +0000 UTC m=+24.980270949" watchObservedRunningTime="2026-07-10 19:08:04.942142529 +0000 UTC m=+24.981015546" Jul 10 19:08:04.958031 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:04.957991 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sxcqz" podStartSLOduration=3.9415368969999998 podStartE2EDuration="23.957975326s" totalImagesPullingTime="20.016438429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:04.957646322 +0000 UTC m=+24.996519339" watchObservedRunningTime="2026-07-10 19:08:04.957975326 +0000 UTC m=+24.996848344" Jul 10 19:08:05.004018 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.003991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:05.007330 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.007293 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-221.ec2.internal" podStartSLOduration=23.007278184 podStartE2EDuration="23.007278184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:05.006312904 +0000 UTC m=+25.045185961" watchObservedRunningTime="2026-07-10 19:08:05.007278184 +0000 UTC m=+25.046151201" Jul 10 19:08:05.098577 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.098548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:05.172042 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.171960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:05.272860 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.272834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:05.573890 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.573836 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 19:08:05.876621 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.876586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" event={"ID":"794cdb83c2901bbdd19f7c4480e67c66","Type":"ContainerStarted","Data":"b535b73bd059df133d3e84056d51eb3141e14d8b4fedeaebc51893bef33decc6"} Jul 10 19:08:05.878567 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.878537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" event={"ID":"159899a6-98e8-4ceb-9c8a-2b110bdf377e","Type":"ContainerStarted","Data":"422f400a48abb238acf4839fbf77d6d171f15360c26fa144852d0cbd92482c34"} Jul 10 19:08:05.893481 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.893426 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-221.ec2.internal" podStartSLOduration=23.893404498 podStartE2EDuration="23.893404498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:05.891716045 +0000 UTC m=+25.930589062" watchObservedRunningTime="2026-07-10 19:08:05.893404498 +0000 UTC m=+25.932277518" Jul 10 19:08:05.894101 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.894003 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T19:08:05.573860019Z","UUID":"95826824-ccf7-492a-8da0-3575b03fa58c","Handler":null,"Name":"","Endpoint":""} Jul 10 19:08:05.896092 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.896072 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 19:08:05.896195 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.896121 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 19:08:05.942204 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:05.942178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:06.007156 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.007126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:06.065475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.065447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:06.065605 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.065448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:06.065605 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:06.065543 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:06.065750 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.065448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:06.065750 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:06.065626 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:06.065750 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:06.065718 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:06.341990 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.341954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:06.445249 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.445215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:06.654210 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.654175 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:06.716914 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.716887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:06.883585 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.883568 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:08:06.884001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:06.883976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"1177f54081d86bf05035c83e6ef18f4d8c3dde2e35662ff4da55154cad770c5e"} Jul 10 19:08:07.169929 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.169852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:07.540930 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.540856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:07.567204 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.567173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:08:07.567779 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.567759 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:08:07.598380 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.598351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:07.888588 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.888549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" event={"ID":"159899a6-98e8-4ceb-9c8a-2b110bdf377e","Type":"ContainerStarted","Data":"e56f36228def9bd92c80548a4c343922425b43c8dfe5604baa347b817aca97db"} Jul 10 19:08:07.888790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.888645 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:08:07.889213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.889194 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cj2bx" Jul 10 19:08:07.907315 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.907256 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zr74k" podStartSLOduration=4.607203352 podStartE2EDuration="26.907240073s" totalImagesPullingTime="22.300036721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:07.906710566 +0000 UTC m=+27.945583585" watchObservedRunningTime="2026-07-10 19:08:07.907240073 +0000 UTC m=+27.946113091" Jul 10 19:08:07.987000 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:07.986956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:08.065042 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:08.064978 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:08.065206 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:08.065089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:08.065206 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:08.065103 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:08.065206 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:08.065119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:08.065370 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:08.065202 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:08.065370 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:08.065276 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:09.486666 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:09.486629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:09.587783 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:09.587759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:09.893917 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:09.893852 2579 generic.go:356] "Generic (PLEG): container finished" podID="1309dd30-f208-401e-a5e0-b33b8a018b90" containerID="7e549d2b540d1323e2cacdc0a209b9275c16330ad1bb1991e85bd683952c3630" exitCode=0 Jul 10 19:08:09.894064 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:09.893947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerDied","Data":"7e549d2b540d1323e2cacdc0a209b9275c16330ad1bb1991e85bd683952c3630"} Jul 10 19:08:09.897530 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:09.897516 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:08:09.897921 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:09.897880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"482150a27ee156287318426665602383c9fd8e4508e8e8d4cc386cf8ac7ab741"} Jul 10 19:08:09.898294 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:09.898279 2579 scope.go:117] "RemoveContainer" containerID="a364d1186336d6b087e42e3cb1d3888f19035f1db140a7d43dd265529ff43082" Jul 10 19:08:10.064850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.064817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:10.064989 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.064857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:10.064989 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.064927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:10.064989 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:10.064928 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:10.065153 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:10.065041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:10.065203 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:10.065138 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:10.903037 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.903006 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:08:10.903991 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.903373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" event={"ID":"1a153daf-5790-42c6-9c33-98f40f104f91","Type":"ContainerStarted","Data":"62e4d1b8878608abcf145f957a9533a0804a225f629310edf02c98893334af06"} Jul 10 19:08:10.903991 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.903673 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:08:10.903991 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.903748 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:08:10.903991 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.903793 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:08:10.919185 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.919166 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:08:10.920313 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.920295 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:08:10.933169 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:10.933125 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" podStartSLOduration=9.601768917 podStartE2EDuration="29.933106665s" totalImagesPullingTime="20.331337748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:10.931102587 +0000 UTC m=+30.969975605" watchObservedRunningTime="2026-07-10 19:08:10.933106665 +0000 UTC m=+30.971979685" Jul 10 19:08:11.059349 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.059322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:11.412874 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.412835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zkfcv"] Jul 10 19:08:11.413005 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.412935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:11.413051 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:11.413016 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:11.413372 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.413340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ssw5f"] Jul 10 19:08:11.413438 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.413427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:11.413511 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:11.413497 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:11.418565 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.418539 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xxz79"] Jul 10 19:08:11.418711 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.418697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:11.418822 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:11.418803 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:11.907182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.907148 2579 generic.go:356] "Generic (PLEG): container finished" podID="1309dd30-f208-401e-a5e0-b33b8a018b90" containerID="2cd27a2bd897557fe3860aeec0b2612d31ebc810c7d410d2b743c87f63e74bb2" exitCode=0 Jul 10 19:08:11.907877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:11.907237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerDied","Data":"2cd27a2bd897557fe3860aeec0b2612d31ebc810c7d410d2b743c87f63e74bb2"} Jul 10 19:08:13.065769 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:13.065529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:13.065769 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:13.065677 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:13.066189 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:13.065839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:13.066189 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:13.065850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:13.066189 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:13.065950 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:13.066189 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:13.066032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:13.862688 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:13.862648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:13.914320 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:13.914294 2579 generic.go:356] "Generic (PLEG): container finished" podID="1309dd30-f208-401e-a5e0-b33b8a018b90" containerID="3fa2e4372ef7a22882d6940684a9fa320b7ab54d43ffdb062226a3452c8fd064" exitCode=0 Jul 10 19:08:13.914479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:13.914329 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerDied","Data":"3fa2e4372ef7a22882d6940684a9fa320b7ab54d43ffdb062226a3452c8fd064"} Jul 10 19:08:13.986558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:13.986533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:14.047432 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:14.047412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:14.380391 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:14.380368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:14.380789 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:14.380403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:14.380789 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.380483 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:08:14.380789 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.380527 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret podName:d1acace1-4b9b-402d-a6ab-f8d8f3a401cc nodeName:}" failed. No retries permitted until 2026-07-10 19:08:46.380514103 +0000 UTC m=+66.419387098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret") pod "global-pull-secret-syncer-ssw5f" (UID: "d1acace1-4b9b-402d-a6ab-f8d8f3a401cc") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:08:14.380789 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.380526 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:08:14.380789 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.380584 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs podName:226ca90d-820d-4c55-b7b2-7f8099397399 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:46.380565381 +0000 UTC m=+66.419438381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs") pod "network-metrics-daemon-xxz79" (UID: "226ca90d-820d-4c55-b7b2-7f8099397399") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:08:14.480728 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:14.480702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:14.480872 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.480828 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:08:14.480872 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.480843 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:08:14.480872 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.480852 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kl5wj for pod openshift-network-diagnostics/network-check-target-zkfcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:08:14.480986 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:14.480897 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj podName:8f760c7e-ca57-4789-a78b-327fdec3f6b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:46.480880589 +0000 UTC m=+66.519753583 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kl5wj" (UniqueName: "kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj") pod "network-check-target-zkfcv" (UID: "8f760c7e-ca57-4789-a78b-327fdec3f6b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:08:15.065639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.065609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:15.065796 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.065609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:15.065796 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:15.065748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkfcv" podUID="8f760c7e-ca57-4789-a78b-327fdec3f6b3" Jul 10 19:08:15.065796 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.065775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:15.065958 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:15.065857 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ssw5f" podUID="d1acace1-4b9b-402d-a6ab-f8d8f3a401cc" Jul 10 19:08:15.066016 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:15.065956 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxz79" podUID="226ca90d-820d-4c55-b7b2-7f8099397399" Jul 10 19:08:15.068238 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.068212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.154363 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.154297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.215432 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.215406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.399788 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.399760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.476480 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.476384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.577523 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.577494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.678750 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.678713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.779797 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:15.779716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:16.178260 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.178229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:16.275472 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.275429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:16.711767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.711682 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-221.ec2.internal" event="NodeReady" Jul 10 19:08:16.712205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.711816 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 19:08:16.751056 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.751027 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666d58b98c-zz945"] Jul 10 19:08:16.753146 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.753106 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.756255 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.756029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 19:08:16.756380 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.756197 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 19:08:16.756380 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.756371 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lqfvm\"" Jul 10 19:08:16.756527 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.756487 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 19:08:16.764819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.764261 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 19:08:16.766095 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.766074 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8pfjr"] Jul 10 19:08:16.768465 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.768446 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6v88z"] Jul 10 19:08:16.768605 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.768582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:16.770373 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.770356 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666d58b98c-zz945"] Jul 10 19:08:16.770468 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.770443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:16.771783 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.771744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z5rf9\"" Jul 10 19:08:16.771876 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.771794 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 19:08:16.771972 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.771798 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 19:08:16.773352 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.773336 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 19:08:16.773550 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.773530 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 19:08:16.773729 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.773626 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 19:08:16.773729 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.773695 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vnt9v\"" Jul 10 19:08:16.773842 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.773759 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 19:08:16.781052 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.781032 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8pfjr"] Jul 10 19:08:16.781781 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.781763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6v88z"] Jul 10 19:08:16.787707 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.787666 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n85lh"] Jul 10 19:08:16.789408 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.789392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:16.792708 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.792621 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 19:08:16.792809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.792738 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4kdxh\"" Jul 10 19:08:16.792809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.792739 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 19:08:16.792933 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.792741 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 19:08:16.801534 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.801434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n85lh"] Jul 10 19:08:16.898414 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-image-registry-private-configuration\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.898580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:16.898580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgkq\" (UniqueName: \"kubernetes.io/projected/c9364374-b9ef-440a-af41-696eb14832c8-kube-api-access-hfgkq\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:16.898580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa44e407-2dce-40f8-9886-a9d779b16f6c-tmp-dir\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:16.898580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c49025d-6cfa-4ef3-ba99-40a5ccc707ab-cert\") pod \"ingress-canary-n85lh\" (UID: \"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab\") " pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:16.898580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.898580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-trusted-ca\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-installation-pull-secrets\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9364374-b9ef-440a-af41-696eb14832c8-data-volume\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsxt\" (UniqueName: \"kubernetes.io/projected/fa44e407-2dce-40f8-9886-a9d779b16f6c-kube-api-access-zxsxt\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40aec87c-18c7-4d15-af90-4f86885b6464-ca-trust-extracted\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-bound-sa-token\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8rv\" (UniqueName: \"kubernetes.io/projected/9c49025d-6cfa-4ef3-ba99-40a5ccc707ab-kube-api-access-cp8rv\") pod \"ingress-canary-n85lh\" (UID: \"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab\") " pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:16.898913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-registry-certificates\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.899294 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44e407-2dce-40f8-9886-a9d779b16f6c-config-volume\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:16.899294 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9364374-b9ef-440a-af41-696eb14832c8-crio-socket\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:16.899294 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.898993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:16.899294 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.899019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696qg\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-kube-api-access-696qg\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:16.903752 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.903728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:17.000077 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:16.999996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-bound-sa-token\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.000077 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8rv\" (UniqueName: \"kubernetes.io/projected/9c49025d-6cfa-4ef3-ba99-40a5ccc707ab-kube-api-access-cp8rv\") pod \"ingress-canary-n85lh\" (UID: \"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab\") " pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:17.000268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-registry-certificates\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.000268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44e407-2dce-40f8-9886-a9d779b16f6c-config-volume\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.000268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9364374-b9ef-440a-af41-696eb14832c8-crio-socket\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.000268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.000268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696qg\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-kube-api-access-696qg\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.000268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-image-registry-private-configuration\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgkq\" (UniqueName: \"kubernetes.io/projected/c9364374-b9ef-440a-af41-696eb14832c8-kube-api-access-hfgkq\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa44e407-2dce-40f8-9886-a9d779b16f6c-tmp-dir\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c49025d-6cfa-4ef3-ba99-40a5ccc707ab-cert\") pod \"ingress-canary-n85lh\" (UID: \"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab\") " pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-trusted-ca\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-installation-pull-secrets\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9364374-b9ef-440a-af41-696eb14832c8-data-volume\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsxt\" (UniqueName: \"kubernetes.io/projected/fa44e407-2dce-40f8-9886-a9d779b16f6c-kube-api-access-zxsxt\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.000553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40aec87c-18c7-4d15-af90-4f86885b6464-ca-trust-extracted\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.001088 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.000752 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:08:17.001088 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.000820 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:17.500800402 +0000 UTC m=+37.539673415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : configmap "kube-rbac-proxy" not found Jul 10 19:08:17.001088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000950 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40aec87c-18c7-4d15-af90-4f86885b6464-ca-trust-extracted\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.001088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.001032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-registry-certificates\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.001088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.001034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa44e407-2dce-40f8-9886-a9d779b16f6c-tmp-dir\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.001088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.000436 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9364374-b9ef-440a-af41-696eb14832c8-crio-socket\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.001088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.001072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44e407-2dce-40f8-9886-a9d779b16f6c-config-volume\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.001107 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.001180 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls podName:fa44e407-2dce-40f8-9886-a9d779b16f6c nodeName:}" failed. No retries permitted until 2026-07-10 19:08:17.501161611 +0000 UTC m=+37.540034640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls") pod "dns-default-8pfjr" (UID: "fa44e407-2dce-40f8-9886-a9d779b16f6c") : secret "dns-default-metrics-tls" not found Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.001239 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.001295 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:17.501278008 +0000 UTC m=+37.540151036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : secret "insights-runtime-extractor-tls" not found Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.001302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9364374-b9ef-440a-af41-696eb14832c8-data-volume\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.001386 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.001405 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d58b98c-zz945: secret "image-registry-tls" not found Jul 10 19:08:17.001493 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.001468 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls podName:40aec87c-18c7-4d15-af90-4f86885b6464 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:17.501449723 +0000 UTC m=+37.540322763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls") pod "image-registry-666d58b98c-zz945" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464") : secret "image-registry-tls" not found Jul 10 19:08:17.002290 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.002260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-trusted-ca\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.004717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.004693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:17.005364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.005346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-image-registry-private-configuration\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.005456 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.005392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-installation-pull-secrets\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.005456 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.005420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c49025d-6cfa-4ef3-ba99-40a5ccc707ab-cert\") pod \"ingress-canary-n85lh\" (UID: \"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab\") " pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:17.009800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.009778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-bound-sa-token\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.010351 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.010316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8rv\" (UniqueName: \"kubernetes.io/projected/9c49025d-6cfa-4ef3-ba99-40a5ccc707ab-kube-api-access-cp8rv\") pod \"ingress-canary-n85lh\" (UID: \"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab\") " pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:17.010573 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.010539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgkq\" (UniqueName: \"kubernetes.io/projected/c9364374-b9ef-440a-af41-696eb14832c8-kube-api-access-hfgkq\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.011048 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.011026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-696qg\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-kube-api-access-696qg\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.011332 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.011293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsxt\" (UniqueName: \"kubernetes.io/projected/fa44e407-2dce-40f8-9886-a9d779b16f6c-kube-api-access-zxsxt\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.065128 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.065102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:17.065247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.065108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:17.065361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.065342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:17.068381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.068109 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 19:08:17.068381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.068121 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwfmf\"" Jul 10 19:08:17.068381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.068143 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 19:08:17.068381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.068206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 19:08:17.068381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.068113 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 19:08:17.068682 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.068474 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-55ch6\"" Jul 10 19:08:17.098951 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.098929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n85lh" Jul 10 19:08:17.471182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.471152 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sxcqz_54573428-5115-497b-9056-2ebd4783c0a5/dns-node-resolver/0.log" Jul 10 19:08:17.505457 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.505426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.505606 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.505537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:17.505606 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.505578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:17.505749 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505599 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:08:17.505749 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505688 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:18.505648227 +0000 UTC m=+38.544521239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : secret "insights-runtime-extractor-tls" not found Jul 10 19:08:17.505749 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505696 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:08:17.505749 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.505605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:17.505949 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505705 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:08:17.505949 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505710 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d58b98c-zz945: secret "image-registry-tls" not found Jul 10 19:08:17.505949 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505828 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls podName:fa44e407-2dce-40f8-9886-a9d779b16f6c nodeName:}" failed. No retries permitted until 2026-07-10 19:08:18.505807442 +0000 UTC m=+38.544680441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls") pod "dns-default-8pfjr" (UID: "fa44e407-2dce-40f8-9886-a9d779b16f6c") : secret "dns-default-metrics-tls" not found Jul 10 19:08:17.505949 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505715 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:08:17.505949 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505886 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls podName:40aec87c-18c7-4d15-af90-4f86885b6464 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:18.505867949 +0000 UTC m=+38.544740968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls") pod "image-registry-666d58b98c-zz945" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464") : secret "image-registry-tls" not found Jul 10 19:08:17.506201 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:17.505957 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:18.505939778 +0000 UTC m=+38.544812788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : configmap "kube-rbac-proxy" not found Jul 10 19:08:17.539969 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.539938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:17.639395 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.639361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:17.738782 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.738714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:17.792508 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.792478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:17.897335 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.897302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:17.953233 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:17.953204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:18.272489 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.272462 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfwrf_c5a5c819-e84a-4d88-8c7a-8be2fa966e33/node-ca/0.log" Jul 10 19:08:18.365139 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.364989 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n85lh"] Jul 10 19:08:18.367578 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:18.367552 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c49025d_6cfa_4ef3_ba99_40a5ccc707ab.slice/crio-108abf3953951a8080bb7a357bc86cb162c71449d9bf3e588dfcf431d35958ea WatchSource:0}: Error finding container 108abf3953951a8080bb7a357bc86cb162c71449d9bf3e588dfcf431d35958ea: Status 404 returned error can't find the container with id 108abf3953951a8080bb7a357bc86cb162c71449d9bf3e588dfcf431d35958ea Jul 10 19:08:18.514344 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.514294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:18.514464 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514445 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:08:18.514499 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.514465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:18.514499 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.514494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:18.514556 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514519 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:20.51450275 +0000 UTC m=+40.553375746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : secret "insights-runtime-extractor-tls" not found Jul 10 19:08:18.514556 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514548 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:08:18.514635 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.514560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:18.514635 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514595 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:08:18.514635 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514598 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:20.51458181 +0000 UTC m=+40.553454817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : configmap "kube-rbac-proxy" not found Jul 10 19:08:18.514766 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514636 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:08:18.514766 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514648 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d58b98c-zz945: secret "image-registry-tls" not found Jul 10 19:08:18.514766 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514682 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls podName:fa44e407-2dce-40f8-9886-a9d779b16f6c nodeName:}" failed. No retries permitted until 2026-07-10 19:08:20.51466897 +0000 UTC m=+40.553541965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls") pod "dns-default-8pfjr" (UID: "fa44e407-2dce-40f8-9886-a9d779b16f6c") : secret "dns-default-metrics-tls" not found Jul 10 19:08:18.514766 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:18.514704 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls podName:40aec87c-18c7-4d15-af90-4f86885b6464 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:20.514691195 +0000 UTC m=+40.553564226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls") pod "image-registry-666d58b98c-zz945" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464") : secret "image-registry-tls" not found Jul 10 19:08:18.646382 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.646358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:18.929075 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.929049 2579 generic.go:356] "Generic (PLEG): container finished" podID="1309dd30-f208-401e-a5e0-b33b8a018b90" containerID="edf94b3c8285310a25bd7f1e7a4baf56889f342394335138c4b34151a822a249" exitCode=0 Jul 10 19:08:18.929841 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.929111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerDied","Data":"edf94b3c8285310a25bd7f1e7a4baf56889f342394335138c4b34151a822a249"} Jul 10 19:08:18.930230 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:18.930211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n85lh" event={"ID":"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab","Type":"ContainerStarted","Data":"108abf3953951a8080bb7a357bc86cb162c71449d9bf3e588dfcf431d35958ea"} Jul 10 19:08:19.267199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:19.267126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:19.935675 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:19.935629 2579 generic.go:356] "Generic (PLEG): container finished" podID="1309dd30-f208-401e-a5e0-b33b8a018b90" containerID="f4717d5542e83ca79db318d1f4cc944a1f9e1d449eca7a51f142b7e8f3770ac7" exitCode=0 Jul 10 19:08:19.935675 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:19.935678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerDied","Data":"f4717d5542e83ca79db318d1f4cc944a1f9e1d449eca7a51f142b7e8f3770ac7"} Jul 10 19:08:20.532874 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:20.532834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:20.533088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:20.532890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:20.533088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:20.532923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:20.533088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:20.532959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:20.533088 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:20.532991 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:08:20.533088 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:20.533054 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls podName:fa44e407-2dce-40f8-9886-a9d779b16f6c nodeName:}" failed. No retries permitted until 2026-07-10 19:08:24.53303704 +0000 UTC m=+44.571910055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls") pod "dns-default-8pfjr" (UID: "fa44e407-2dce-40f8-9886-a9d779b16f6c") : secret "dns-default-metrics-tls" not found Jul 10 19:08:20.533332 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:20.533120 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:08:20.533332 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:20.533149 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:08:20.533332 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:20.533162 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d58b98c-zz945: secret "image-registry-tls" not found Jul 10 19:08:20.533332 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:20.533193 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:24.53317413 +0000 UTC m=+44.572047145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : secret "insights-runtime-extractor-tls" not found Jul 10 19:08:20.533332 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:20.533215 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls podName:40aec87c-18c7-4d15-af90-4f86885b6464 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:24.533203133 +0000 UTC m=+44.572076135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls") pod "image-registry-666d58b98c-zz945" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464") : secret "image-registry-tls" not found Jul 10 19:08:20.533630 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:20.533607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9364374-b9ef-440a-af41-696eb14832c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:20.940112 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:20.940084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4r25b" event={"ID":"1309dd30-f208-401e-a5e0-b33b8a018b90","Type":"ContainerStarted","Data":"ebab2a476a783f575c14b7587571708ac52f126c16d0281730bccccd94bc332e"} Jul 10 19:08:20.962984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:20.962940 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4r25b" podStartSLOduration=8.617352628999999 podStartE2EDuration="39.962925433s" totalImagesPullingTime="31.345572804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:20.961048049 +0000 UTC m=+40.999921067" watchObservedRunningTime="2026-07-10 19:08:20.962925433 +0000 UTC m=+41.001798447" Jul 10 19:08:21.103154 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:21.103127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:21.424029 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:21.423866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:21.661596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:21.661571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:21.762828 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:21.762807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:21.944000 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:21.943941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n85lh" event={"ID":"9c49025d-6cfa-4ef3-ba99-40a5ccc707ab","Type":"ContainerStarted","Data":"5e66a51cf4b5594ed5436553b60ba818fcc2a628709a23e49648c21e79569ba2"} Jul 10 19:08:21.960180 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:21.960044 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n85lh" podStartSLOduration=3.200747118 podStartE2EDuration="5.960028992s" totalImagesPullingTime="2.759281874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:21.959705493 +0000 UTC m=+41.998578528" watchObservedRunningTime="2026-07-10 19:08:21.960028992 +0000 UTC m=+41.998902009" Jul 10 19:08:23.116406 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:23.116348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:23.358367 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:23.358336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:23.459735 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:23.459670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:23.560142 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:23.560118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:23.665323 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:23.665290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:23.761102 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:23.761041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:23.941164 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:23.941135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:24.562337 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:24.562300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:24.562337 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:24.562343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:24.562442 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:24.562458 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666d58b98c-zz945: secret "image-registry-tls" not found Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:24.562506 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls podName:40aec87c-18c7-4d15-af90-4f86885b6464 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:32.562491945 +0000 UTC m=+52.601364940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls") pod "image-registry-666d58b98c-zz945" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464") : secret "image-registry-tls" not found Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:24.562518 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:24.562562 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls podName:c9364374-b9ef-440a-af41-696eb14832c8 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:32.562547704 +0000 UTC m=+52.601420699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6v88z" (UID: "c9364374-b9ef-440a-af41-696eb14832c8") : secret "insights-runtime-extractor-tls" not found Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:24.562626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:24.562719 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:08:24.562767 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:24.562747 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls podName:fa44e407-2dce-40f8-9886-a9d779b16f6c nodeName:}" failed. No retries permitted until 2026-07-10 19:08:32.562739899 +0000 UTC m=+52.601612893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls") pod "dns-default-8pfjr" (UID: "fa44e407-2dce-40f8-9886-a9d779b16f6c") : secret "dns-default-metrics-tls" not found Jul 10 19:08:25.135013 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:25.134988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:25.332126 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:25.332100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:25.569944 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:25.569877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:25.731196 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:25.731163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:25.832114 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:25.832054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:25.934602 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:25.934568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:26.801431 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:26.801390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:26.934368 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:26.934341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:27.036140 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:27.036112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:27.334299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:27.334271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:27.531440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:27.531410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:27.734276 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:27.734194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:27.835979 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:27.835951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:28.130065 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:28.130036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:28.736213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:28.736185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:28.819071 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:28.819029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:28.877234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:28.877209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:29.137163 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:29.137140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:29.338687 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:29.338648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:29.564985 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:29.564914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:29.731354 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:29.731318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:30.131504 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:30.131471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:30.307069 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:30.307039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:30.533423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:30.533352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:31.334508 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:31.334478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:31.531454 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:31.531415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:31.933136 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:31.933106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:32.135015 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.134987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:32.568786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.568760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:32.621925 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.621902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:32.621997 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.621935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:32.621997 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.621989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:32.625720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.625692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa44e407-2dce-40f8-9886-a9d779b16f6c-metrics-tls\") pod \"dns-default-8pfjr\" (UID: \"fa44e407-2dce-40f8-9886-a9d779b16f6c\") " pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:32.625818 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.625743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9364374-b9ef-440a-af41-696eb14832c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6v88z\" (UID: \"c9364374-b9ef-440a-af41-696eb14832c8\") " pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:32.625857 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.625838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"image-registry-666d58b98c-zz945\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:32.667826 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.667805 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:32.680085 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.680064 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:32.686336 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.686316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6v88z" Jul 10 19:08:32.837470 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.837433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666d58b98c-zz945"] Jul 10 19:08:32.840067 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:32.840042 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40aec87c_18c7_4d15_af90_4f86885b6464.slice/crio-2cbffb0fa3f32c2db96bcfa3fde5353250aa15f4b8e816a74785e044cf11b83c WatchSource:0}: Error finding container 2cbffb0fa3f32c2db96bcfa3fde5353250aa15f4b8e816a74785e044cf11b83c: Status 404 returned error can't find the container with id 2cbffb0fa3f32c2db96bcfa3fde5353250aa15f4b8e816a74785e044cf11b83c Jul 10 19:08:32.859721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.859700 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8pfjr"] Jul 10 19:08:32.862543 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:32.862521 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa44e407_2dce_40f8_9886_a9d779b16f6c.slice/crio-53496af21128021379c8592e2bd0bcd8141b59e96ed4f7ca5579fd8ce945cc19 WatchSource:0}: Error finding container 53496af21128021379c8592e2bd0bcd8141b59e96ed4f7ca5579fd8ce945cc19: Status 404 returned error can't find the container with id 53496af21128021379c8592e2bd0bcd8141b59e96ed4f7ca5579fd8ce945cc19 Jul 10 19:08:32.881532 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.881506 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6v88z"] Jul 10 19:08:32.885300 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:32.885272 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9364374_b9ef_440a_af41_696eb14832c8.slice/crio-4918379ff30e2aab18ccb556e6217a76612167714b6c8612bf115aaf63f09bf5 WatchSource:0}: Error finding container 4918379ff30e2aab18ccb556e6217a76612167714b6c8612bf115aaf63f09bf5: Status 404 returned error can't find the container with id 4918379ff30e2aab18ccb556e6217a76612167714b6c8612bf115aaf63f09bf5 Jul 10 19:08:32.967541 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.967516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8pfjr" event={"ID":"fa44e407-2dce-40f8-9886-a9d779b16f6c","Type":"ContainerStarted","Data":"53496af21128021379c8592e2bd0bcd8141b59e96ed4f7ca5579fd8ce945cc19"} Jul 10 19:08:32.968608 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.968573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v88z" event={"ID":"c9364374-b9ef-440a-af41-696eb14832c8","Type":"ContainerStarted","Data":"4918379ff30e2aab18ccb556e6217a76612167714b6c8612bf115aaf63f09bf5"} Jul 10 19:08:32.969896 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.969875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d58b98c-zz945" event={"ID":"40aec87c-18c7-4d15-af90-4f86885b6464","Type":"ContainerStarted","Data":"1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b"} Jul 10 19:08:32.969978 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:32.969916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d58b98c-zz945" event={"ID":"40aec87c-18c7-4d15-af90-4f86885b6464","Type":"ContainerStarted","Data":"2cbffb0fa3f32c2db96bcfa3fde5353250aa15f4b8e816a74785e044cf11b83c"} Jul 10 19:08:33.168833 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:33.168762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:33.614333 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:33.614244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:33.741261 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:33.741228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:33.934632 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:33.934530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:33.974238 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:33.974169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:08:33.993914 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:33.993868 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666d58b98c-zz945" podStartSLOduration=26.993852359 podStartE2EDuration="26.993852359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:33.993321715 +0000 UTC m=+54.032194749" watchObservedRunningTime="2026-07-10 19:08:33.993852359 +0000 UTC m=+54.032725367" Jul 10 19:08:36.342294 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:36.342220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:36.441252 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:36.441221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:36.623211 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:36.623129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:36.724983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:36.724956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:36.823509 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:36.823480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:36.879191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:36.879170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:36.998205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:36.998160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:37.103028 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:37.102993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:37.392976 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:37.392947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:37.494292 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:37.494250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:37.986342 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:37.986279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v88z" event={"ID":"c9364374-b9ef-440a-af41-696eb14832c8","Type":"ContainerStarted","Data":"f55d873fb6d8da4f704ed54bdb3d99953481e7adb89bd3c1eb16d654b8eebe29"} Jul 10 19:08:37.988075 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:37.988050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8pfjr" event={"ID":"fa44e407-2dce-40f8-9886-a9d779b16f6c","Type":"ContainerStarted","Data":"eff3ceaaed1fc9917f442e86242a8696080205f79b6d9c6e0a5420d8bf1473b9"} Jul 10 19:08:37.988193 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:37.988097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8pfjr" event={"ID":"fa44e407-2dce-40f8-9886-a9d779b16f6c","Type":"ContainerStarted","Data":"824e62c9a7bc6713ebe5f4eb12c7a026b69bb220366c8ff28e4bc9c9869b60c6"} Jul 10 19:08:37.988193 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:37.988174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:38.005094 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:38.005052 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8pfjr" podStartSLOduration=17.141118882 podStartE2EDuration="22.005038546s" totalImagesPullingTime="4.863919664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:38.003493496 +0000 UTC m=+58.042366512" watchObservedRunningTime="2026-07-10 19:08:38.005038546 +0000 UTC m=+58.043911563" Jul 10 19:08:38.012341 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:38.012314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:38.211140 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:38.211115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:38.525422 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:38.525397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:38.993538 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:38.993503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v88z" event={"ID":"c9364374-b9ef-440a-af41-696eb14832c8","Type":"ContainerStarted","Data":"01f7a2a262cfecff99c34126ed1581302eb3457a764208a9a1a70c66714ed791"} Jul 10 19:08:39.031254 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.031226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:39.132649 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.132614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:39.605475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.605420 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj"] Jul 10 19:08:39.607456 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.607422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:39.613249 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:39.613204 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-221.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-132-221.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 10 19:08:39.613367 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:39.613260 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-dockercfg-cl96t\" is forbidden: User \"system:node:ip-10-0-132-221.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-132-221.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cl96t\"" type="*v1.Secret" Jul 10 19:08:39.613824 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:39.613798 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-132-221.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-132-221.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" Jul 10 19:08:39.614050 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:39.614015 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-221.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-132-221.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 10 19:08:39.614119 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:39.614096 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-132-221.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-132-221.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Jul 10 19:08:39.614690 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:39.614642 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"managed-serviceaccount-addon-agent-5c95cd7678-9f9xj\" is forbidden: User \"system:node:ip-10-0-132-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-132-221.ec2.internal' and this object" podUID="55d9bd02-651f-4d8d-a654-5509d19f5280" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:39.620775 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.620757 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-85xd9"] Jul 10 19:08:39.622702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.622680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.627109 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.627091 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 19:08:39.627191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.627163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 19:08:39.631208 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.630957 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t9brj\"" Jul 10 19:08:39.632797 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.632777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj"] Jul 10 19:08:39.661388 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.661365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-85xd9"] Jul 10 19:08:39.669149 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.669128 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666d58b98c-zz945"] Jul 10 19:08:39.676489 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.676465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xc5\" (UniqueName: \"kubernetes.io/projected/55d9bd02-651f-4d8d-a654-5509d19f5280-kube-api-access-95xc5\") pod \"managed-serviceaccount-addon-agent-5c95cd7678-9f9xj\" (UID: \"55d9bd02-651f-4d8d-a654-5509d19f5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:39.676593 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.676507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55d9bd02-651f-4d8d-a654-5509d19f5280-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c95cd7678-9f9xj\" (UID: \"55d9bd02-651f-4d8d-a654-5509d19f5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:39.676593 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.676550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e68db36c-a90d-481c-9ce5-720c1610e7c4-nginx-conf\") pod \"networking-console-plugin-6cf856758c-85xd9\" (UID: \"e68db36c-a90d-481c-9ce5-720c1610e7c4\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.676593 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.676577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e68db36c-a90d-481c-9ce5-720c1610e7c4-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-85xd9\" (UID: \"e68db36c-a90d-481c-9ce5-720c1610e7c4\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.720354 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.720333 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-fh6rn"] Jul 10 19:08:39.722506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.722490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-fh6rn" Jul 10 19:08:39.726027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.726005 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 19:08:39.726138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.726053 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2zgr2\"" Jul 10 19:08:39.726383 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.726366 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 19:08:39.738258 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.738212 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-fh6rn"] Jul 10 19:08:39.777409 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.777368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e68db36c-a90d-481c-9ce5-720c1610e7c4-nginx-conf\") pod \"networking-console-plugin-6cf856758c-85xd9\" (UID: \"e68db36c-a90d-481c-9ce5-720c1610e7c4\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.777531 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.777415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e68db36c-a90d-481c-9ce5-720c1610e7c4-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-85xd9\" (UID: \"e68db36c-a90d-481c-9ce5-720c1610e7c4\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.777531 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.777473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddhg\" (UniqueName: \"kubernetes.io/projected/a7b4d22f-9d66-4edc-b47d-e60671ac5979-kube-api-access-zddhg\") pod \"downloads-68c9cbb497-fh6rn\" (UID: \"a7b4d22f-9d66-4edc-b47d-e60671ac5979\") " pod="openshift-console/downloads-68c9cbb497-fh6rn" Jul 10 19:08:39.777631 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.777535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xc5\" (UniqueName: \"kubernetes.io/projected/55d9bd02-651f-4d8d-a654-5509d19f5280-kube-api-access-95xc5\") pod \"managed-serviceaccount-addon-agent-5c95cd7678-9f9xj\" (UID: \"55d9bd02-651f-4d8d-a654-5509d19f5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:39.777631 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.777553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55d9bd02-651f-4d8d-a654-5509d19f5280-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c95cd7678-9f9xj\" (UID: \"55d9bd02-651f-4d8d-a654-5509d19f5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:39.778130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.778112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e68db36c-a90d-481c-9ce5-720c1610e7c4-nginx-conf\") pod \"networking-console-plugin-6cf856758c-85xd9\" (UID: \"e68db36c-a90d-481c-9ce5-720c1610e7c4\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.780310 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.780287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e68db36c-a90d-481c-9ce5-720c1610e7c4-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-85xd9\" (UID: \"e68db36c-a90d-481c-9ce5-720c1610e7c4\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.787334 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.787301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:39.808156 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.808136 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75fb498fbd-hgkfj"] Jul 10 19:08:39.810318 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.810302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.823046 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.823026 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fb498fbd-hgkfj"] Jul 10 19:08:39.878247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddhg\" (UniqueName: \"kubernetes.io/projected/a7b4d22f-9d66-4edc-b47d-e60671ac5979-kube-api-access-zddhg\") pod \"downloads-68c9cbb497-fh6rn\" (UID: \"a7b4d22f-9d66-4edc-b47d-e60671ac5979\") " pod="openshift-console/downloads-68c9cbb497-fh6rn" Jul 10 19:08:39.878364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-registry-tls\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.878364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-trusted-ca\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.878364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-image-registry-private-configuration\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.878364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-ca-trust-extracted\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.878364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-registry-certificates\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.878546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2mk\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-kube-api-access-jj2mk\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.878546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-installation-pull-secrets\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.878546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.878463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-bound-sa-token\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.888775 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.888742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:39.889083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.889066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddhg\" (UniqueName: \"kubernetes.io/projected/a7b4d22f-9d66-4edc-b47d-e60671ac5979-kube-api-access-zddhg\") pod \"downloads-68c9cbb497-fh6rn\" (UID: \"a7b4d22f-9d66-4edc-b47d-e60671ac5979\") " pod="openshift-console/downloads-68c9cbb497-fh6rn" Jul 10 19:08:39.934197 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.934172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" Jul 10 19:08:39.978822 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.978797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-registry-tls\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.978938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.978837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-trusted-ca\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.978938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.978865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-image-registry-private-configuration\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.978938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.978884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-ca-trust-extracted\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.979087 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.979000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-registry-certificates\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.979087 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.979041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2mk\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-kube-api-access-jj2mk\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.979191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.979093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-installation-pull-secrets\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.979191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.979131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-bound-sa-token\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.979287 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.979239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-ca-trust-extracted\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.979953 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.979916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-trusted-ca\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.980124 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.980100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-registry-certificates\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.981912 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.981888 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-image-registry-private-configuration\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.982004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.981967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-registry-tls\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.982213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.982189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-installation-pull-secrets\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.989909 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.989881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:39.990335 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.990313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-bound-sa-token\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:39.992170 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:39.992148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2mk\" (UniqueName: \"kubernetes.io/projected/1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3-kube-api-access-jj2mk\") pod \"image-registry-75fb498fbd-hgkfj\" (UID: \"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3\") " pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:40.032518 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.032491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-fh6rn" Jul 10 19:08:40.093982 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.093956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:40.120436 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.120408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:40.186804 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.186740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:40.287244 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.287213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:40.423926 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.423895 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 19:08:40.453271 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.453210 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 19:08:40.460983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.460960 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55d9bd02-651f-4d8d-a654-5509d19f5280-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c95cd7678-9f9xj\" (UID: \"55d9bd02-651f-4d8d-a654-5509d19f5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:40.706488 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.706401 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cl96t\"" Jul 10 19:08:40.721528 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:40.721502 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 19:08:41.070023 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:41.069998 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 19:08:41.078640 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:41.078605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xc5\" (UniqueName: \"kubernetes.io/projected/55d9bd02-651f-4d8d-a654-5509d19f5280-kube-api-access-95xc5\") pod \"managed-serviceaccount-addon-agent-5c95cd7678-9f9xj\" (UID: \"55d9bd02-651f-4d8d-a654-5509d19f5280\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:41.096522 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:41.096502 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fb498fbd-hgkfj"] Jul 10 19:08:41.098608 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:41.098583 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7665b8_2fb9_4e5f_93f9_bb1a7bd166a3.slice/crio-d69010d5d963c7fa2d4431b56d6307b18fe159c73cc4554863d9b1bbc1e7d77a WatchSource:0}: Error finding container d69010d5d963c7fa2d4431b56d6307b18fe159c73cc4554863d9b1bbc1e7d77a: Status 404 returned error can't find the container with id d69010d5d963c7fa2d4431b56d6307b18fe159c73cc4554863d9b1bbc1e7d77a Jul 10 19:08:41.125182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:41.125145 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" Jul 10 19:08:41.248113 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:41.248046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj"] Jul 10 19:08:41.251299 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:41.251266 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55d9bd02_651f_4d8d_a654_5509d19f5280.slice/crio-b787c044dbaf29d0f7eae2d437a9fdbda17b930032b9231b16fc1f94c7258c50 WatchSource:0}: Error finding container b787c044dbaf29d0f7eae2d437a9fdbda17b930032b9231b16fc1f94c7258c50: Status 404 returned error can't find the container with id b787c044dbaf29d0f7eae2d437a9fdbda17b930032b9231b16fc1f94c7258c50 Jul 10 19:08:41.307179 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:41.307149 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-fh6rn"] Jul 10 19:08:41.310113 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:41.310087 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-85xd9"] Jul 10 19:08:41.310580 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:41.310552 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b4d22f_9d66_4edc_b47d_e60671ac5979.slice/crio-0f3dc416613fc4a48e38ec8ecfed8bf1110f684f5b694d56127dcb0ace330cac WatchSource:0}: Error finding container 0f3dc416613fc4a48e38ec8ecfed8bf1110f684f5b694d56127dcb0ace330cac: Status 404 returned error can't find the container with id 0f3dc416613fc4a48e38ec8ecfed8bf1110f684f5b694d56127dcb0ace330cac Jul 10 19:08:42.009230 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.009190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" event={"ID":"55d9bd02-651f-4d8d-a654-5509d19f5280","Type":"ContainerStarted","Data":"b787c044dbaf29d0f7eae2d437a9fdbda17b930032b9231b16fc1f94c7258c50"} Jul 10 19:08:42.010912 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.010880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-fh6rn" event={"ID":"a7b4d22f-9d66-4edc-b47d-e60671ac5979","Type":"ContainerStarted","Data":"0f3dc416613fc4a48e38ec8ecfed8bf1110f684f5b694d56127dcb0ace330cac"} Jul 10 19:08:42.014748 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.014264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6v88z" event={"ID":"c9364374-b9ef-440a-af41-696eb14832c8","Type":"ContainerStarted","Data":"caa806ef08ae06b54b4203f055721b05218e732b51259b8a3a85e6651e18c070"} Jul 10 19:08:42.017131 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.017047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" event={"ID":"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3","Type":"ContainerStarted","Data":"63e50d82c48484aa1510fa48a4fd31b840c74a331e73b99f7765c065f43f636d"} Jul 10 19:08:42.017131 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.017090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" event={"ID":"1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3","Type":"ContainerStarted","Data":"d69010d5d963c7fa2d4431b56d6307b18fe159c73cc4554863d9b1bbc1e7d77a"} Jul 10 19:08:42.017567 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.017549 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:08:42.018962 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.018908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" event={"ID":"e68db36c-a90d-481c-9ce5-720c1610e7c4","Type":"ContainerStarted","Data":"b3b22f3437d0884bf3a2f82f1cf1984b852b662cc72bc3974c2da60d2cc79e42"} Jul 10 19:08:42.035323 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.035276 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6v88z" podStartSLOduration=18.053096781 podStartE2EDuration="26.035258923s" totalImagesPullingTime="7.982162142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:08:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:42.033712612 +0000 UTC m=+62.072585629" watchObservedRunningTime="2026-07-10 19:08:42.035258923 +0000 UTC m=+62.074131942" Jul 10 19:08:42.929474 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:42.929442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tm9q6" Jul 10 19:08:43.001338 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.001287 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" podStartSLOduration=4.001272709 podStartE2EDuration="4.001272709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:42.05415637 +0000 UTC m=+62.093029368" watchObservedRunningTime="2026-07-10 19:08:43.001272709 +0000 UTC m=+63.040145704" Jul 10 19:08:43.204374 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.204300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:43.308202 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.308172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:43.402292 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.402257 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v"] Jul 10 19:08:43.408974 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.408908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:43.437841 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.437806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v"] Jul 10 19:08:43.437981 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.437904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" Jul 10 19:08:43.442059 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.442034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 19:08:43.442176 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.442034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2kjdh\"" Jul 10 19:08:43.491133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.491079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:43.509296 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.509272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c90b0659-9974-4e23-92d8-351dccb06ce0-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-dpf2v\" (UID: \"c90b0659-9974-4e23-92d8-351dccb06ce0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" Jul 10 19:08:43.610780 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.610748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c90b0659-9974-4e23-92d8-351dccb06ce0-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-dpf2v\" (UID: \"c90b0659-9974-4e23-92d8-351dccb06ce0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" Jul 10 19:08:43.613494 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.613466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c90b0659-9974-4e23-92d8-351dccb06ce0-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-dpf2v\" (UID: \"c90b0659-9974-4e23-92d8-351dccb06ce0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" Jul 10 19:08:43.750423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:43.750349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" Jul 10 19:08:44.525026 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:44.525000 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v"] Jul 10 19:08:44.529495 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:44.529056 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90b0659_9974_4e23_92d8_351dccb06ce0.slice/crio-4298978881d6f714633d6f515acfd1654ad2c3190af0230b9b665e5f4c549df2 WatchSource:0}: Error finding container 4298978881d6f714633d6f515acfd1654ad2c3190af0230b9b665e5f4c549df2: Status 404 returned error can't find the container with id 4298978881d6f714633d6f515acfd1654ad2c3190af0230b9b665e5f4c549df2 Jul 10 19:08:45.032975 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:45.032937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" event={"ID":"c90b0659-9974-4e23-92d8-351dccb06ce0","Type":"ContainerStarted","Data":"4298978881d6f714633d6f515acfd1654ad2c3190af0230b9b665e5f4c549df2"} Jul 10 19:08:45.034665 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:45.034631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" event={"ID":"55d9bd02-651f-4d8d-a654-5509d19f5280","Type":"ContainerStarted","Data":"88f86173462d1b4ad2df2112e5a2e203d0cbcbf5d98faddbcd2f2b526c0f2104"} Jul 10 19:08:45.036270 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:45.036246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" event={"ID":"e68db36c-a90d-481c-9ce5-720c1610e7c4","Type":"ContainerStarted","Data":"10498f296457bb6cf3a8a310848bc545cba410f7ebd9af86d4bb853c94796ee8"} Jul 10 19:08:45.051375 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:45.051334 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c95cd7678-9f9xj" podStartSLOduration=2.881748358 podStartE2EDuration="6.051320429s" totalImagesPullingTime="3.169572071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:45.050530732 +0000 UTC m=+65.089403751" watchObservedRunningTime="2026-07-10 19:08:45.051320429 +0000 UTC m=+65.090193445" Jul 10 19:08:45.066954 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:45.066917 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-85xd9" podStartSLOduration=3.005696023 podStartE2EDuration="6.066904455s" totalImagesPullingTime="3.061208432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:45.066784915 +0000 UTC m=+65.105657943" watchObservedRunningTime="2026-07-10 19:08:45.066904455 +0000 UTC m=+65.105777471" Jul 10 19:08:45.182209 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:45.182182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:45.246176 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:45.246144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:46.041286 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.041246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" event={"ID":"c90b0659-9974-4e23-92d8-351dccb06ce0","Type":"ContainerStarted","Data":"0154c58a498907504d3ccdf11abbf349ee83349f4e8b2cdc16c1682399450a5c"} Jul 10 19:08:46.041750 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.041545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" Jul 10 19:08:46.043766 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.043740 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-575cfcd9c6-wm2g6"] Jul 10 19:08:46.046100 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.046080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.047418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.047394 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" Jul 10 19:08:46.048599 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.048569 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 19:08:46.048723 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.048601 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 19:08:46.048823 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.048804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 19:08:46.048918 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.048855 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 19:08:46.049782 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.049763 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 19:08:46.049861 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.049788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9xtw5\"" Jul 10 19:08:46.055271 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.055251 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575cfcd9c6-wm2g6"] Jul 10 19:08:46.056668 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.056613 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-dpf2v" podStartSLOduration=1.753300503 podStartE2EDuration="3.056598953s" totalImagesPullingTime="1.30329845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:46.056482778 +0000 UTC m=+66.095355795" watchObservedRunningTime="2026-07-10 19:08:46.056598953 +0000 UTC m=+66.095471965" Jul 10 19:08:46.132333 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.132294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-oauth-serving-cert\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.132488 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.132358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-console-config\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.132488 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.132386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hltsr\" (UniqueName: \"kubernetes.io/projected/17c1563f-724f-4b56-becd-27544a07e426-kube-api-access-hltsr\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.132488 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.132451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-serving-cert\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.132488 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.132478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-service-ca\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.132711 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.132518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-oauth-config\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.232970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.232943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-oauth-config\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.233115 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.233002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-oauth-serving-cert\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.233115 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.233027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-console-config\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.233115 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.233052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hltsr\" (UniqueName: \"kubernetes.io/projected/17c1563f-724f-4b56-becd-27544a07e426-kube-api-access-hltsr\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.233115 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.233099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-serving-cert\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.233307 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.233124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-service-ca\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.234045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.233998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-oauth-serving-cert\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.234148 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.234060 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-console-config\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.234148 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.234081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-service-ca\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.235793 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.235768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-serving-cert\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.235793 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.235780 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-oauth-config\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.243337 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.243294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hltsr\" (UniqueName: \"kubernetes.io/projected/17c1563f-724f-4b56-becd-27544a07e426-kube-api-access-hltsr\") pod \"console-575cfcd9c6-wm2g6\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.357543 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.357517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:08:46.411845 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.411812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:46.434805 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.434775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:46.434929 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.434869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:46.437629 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.437592 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 19:08:46.437773 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.437745 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 19:08:46.448411 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.448357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226ca90d-820d-4c55-b7b2-7f8099397399-metrics-certs\") pod \"network-metrics-daemon-xxz79\" (UID: \"226ca90d-820d-4c55-b7b2-7f8099397399\") " pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:46.448552 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.448527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1acace1-4b9b-402d-a6ab-f8d8f3a401cc-original-pull-secret\") pod \"global-pull-secret-syncer-ssw5f\" (UID: \"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc\") " pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:46.456384 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.456363 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tnkss"] Jul 10 19:08:46.460795 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.460772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.463666 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.463627 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 19:08:46.463776 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.463679 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 10 19:08:46.464113 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.464091 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 19:08:46.464113 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.464105 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bwrzd\"" Jul 10 19:08:46.464397 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.464350 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 10 19:08:46.464397 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.464363 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 19:08:46.469777 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.469749 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tnkss"] Jul 10 19:08:46.476776 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.476756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ssw5f" Jul 10 19:08:46.487808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.487781 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-55ch6\"" Jul 10 19:08:46.492789 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.492768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575cfcd9c6-wm2g6"] Jul 10 19:08:46.494085 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.494045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:46.494846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.494828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxz79" Jul 10 19:08:46.496224 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:46.496199 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c1563f_724f_4b56_becd_27544a07e426.slice/crio-1f44986066df6fac67de6c2af579409fda085d62601e30f254ad056fe69d9b68 WatchSource:0}: Error finding container 1f44986066df6fac67de6c2af579409fda085d62601e30f254ad056fe69d9b68: Status 404 returned error can't find the container with id 1f44986066df6fac67de6c2af579409fda085d62601e30f254ad056fe69d9b68 Jul 10 19:08:46.536364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.536071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:46.536364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.536147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.536364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.536174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnj5z\" (UniqueName: \"kubernetes.io/projected/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-kube-api-access-gnj5z\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.536364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.536212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.536364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.536256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.539726 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.539257 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 19:08:46.549599 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.549562 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 19:08:46.560374 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.560317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5wj\" (UniqueName: \"kubernetes.io/projected/8f760c7e-ca57-4789-a78b-327fdec3f6b3-kube-api-access-kl5wj\") pod \"network-check-target-zkfcv\" (UID: \"8f760c7e-ca57-4789-a78b-327fdec3f6b3\") " pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:46.624826 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.624789 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ssw5f"] Jul 10 19:08:46.627042 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:46.627016 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1acace1_4b9b_402d_a6ab_f8d8f3a401cc.slice/crio-d5a441be55e93280e7f07b7e41673ecff8b460061857560a1181702e20d1a9a7 WatchSource:0}: Error finding container d5a441be55e93280e7f07b7e41673ecff8b460061857560a1181702e20d1a9a7: Status 404 returned error can't find the container with id d5a441be55e93280e7f07b7e41673ecff8b460061857560a1181702e20d1a9a7 Jul 10 19:08:46.637088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.637067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.637184 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.637095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnj5z\" (UniqueName: \"kubernetes.io/projected/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-kube-api-access-gnj5z\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.637184 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.637118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.637184 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.637137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.637334 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:46.637240 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 10 19:08:46.637334 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:08:46.637299 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-tls podName:8fbf6a90-cbaf-41a7-abc3-de3f54620f0a nodeName:}" failed. No retries permitted until 2026-07-10 19:08:47.137278246 +0000 UTC m=+67.176151255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-tls") pod "prometheus-operator-64785b6675-tnkss" (UID: "8fbf6a90-cbaf-41a7-abc3-de3f54620f0a") : secret "prometheus-operator-tls" not found Jul 10 19:08:46.637846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.637823 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-metrics-client-ca\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.640707 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.640684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.641382 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.641361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xxz79"] Jul 10 19:08:46.644462 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:46.644440 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226ca90d_820d_4c55_b7b2_7f8099397399.slice/crio-f514a91d92f1124626e4da3eefd81def16aa7ec2d3ba6621acdd61c781719d47 WatchSource:0}: Error finding container f514a91d92f1124626e4da3eefd81def16aa7ec2d3ba6621acdd61c781719d47: Status 404 returned error can't find the container with id f514a91d92f1124626e4da3eefd81def16aa7ec2d3ba6621acdd61c781719d47 Jul 10 19:08:46.650301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.650185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnj5z\" (UniqueName: \"kubernetes.io/projected/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-kube-api-access-gnj5z\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:46.691083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.691018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:46.795822 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.795791 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwfmf\"" Jul 10 19:08:46.800881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.800865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:08:46.920846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:46.920824 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zkfcv"] Jul 10 19:08:46.922999 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:46.922976 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f760c7e_ca57_4789_a78b_327fdec3f6b3.slice/crio-c855484db6a95721ca04886c6a70fa3e41ee3ac23829e4f8b5e2f5e202513f6c WatchSource:0}: Error finding container c855484db6a95721ca04886c6a70fa3e41ee3ac23829e4f8b5e2f5e202513f6c: Status 404 returned error can't find the container with id c855484db6a95721ca04886c6a70fa3e41ee3ac23829e4f8b5e2f5e202513f6c Jul 10 19:08:47.047447 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.047374 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ssw5f" event={"ID":"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc","Type":"ContainerStarted","Data":"d5a441be55e93280e7f07b7e41673ecff8b460061857560a1181702e20d1a9a7"} Jul 10 19:08:47.048455 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.048429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zkfcv" event={"ID":"8f760c7e-ca57-4789-a78b-327fdec3f6b3","Type":"ContainerStarted","Data":"c855484db6a95721ca04886c6a70fa3e41ee3ac23829e4f8b5e2f5e202513f6c"} Jul 10 19:08:47.049545 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.049526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxz79" event={"ID":"226ca90d-820d-4c55-b7b2-7f8099397399","Type":"ContainerStarted","Data":"f514a91d92f1124626e4da3eefd81def16aa7ec2d3ba6621acdd61c781719d47"} Jul 10 19:08:47.050570 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.050554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cfcd9c6-wm2g6" event={"ID":"17c1563f-724f-4b56-becd-27544a07e426","Type":"ContainerStarted","Data":"1f44986066df6fac67de6c2af579409fda085d62601e30f254ad056fe69d9b68"} Jul 10 19:08:47.141218 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.141176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:47.143945 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.143920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fbf6a90-cbaf-41a7-abc3-de3f54620f0a-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-tnkss\" (UID: \"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:47.372708 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.372448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" Jul 10 19:08:47.565983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.565320 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-tnkss"] Jul 10 19:08:47.909952 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:08:47.909914 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbf6a90_cbaf_41a7_abc3_de3f54620f0a.slice/crio-250c5d42ea5029c0a7a17410d9a5db4222a65eb593da553720e59e09f3fe83da WatchSource:0}: Error finding container 250c5d42ea5029c0a7a17410d9a5db4222a65eb593da553720e59e09f3fe83da: Status 404 returned error can't find the container with id 250c5d42ea5029c0a7a17410d9a5db4222a65eb593da553720e59e09f3fe83da Jul 10 19:08:47.996607 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:47.996566 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8pfjr" Jul 10 19:08:48.071391 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:48.071319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" event={"ID":"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a","Type":"ContainerStarted","Data":"250c5d42ea5029c0a7a17410d9a5db4222a65eb593da553720e59e09f3fe83da"} Jul 10 19:08:49.081792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.081738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxz79" event={"ID":"226ca90d-820d-4c55-b7b2-7f8099397399","Type":"ContainerStarted","Data":"3023fb790fe50b0c6be56dbb26cb79d7d8a31b790fe224222699f73bee4624ea"} Jul 10 19:08:49.082255 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.081805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxz79" event={"ID":"226ca90d-820d-4c55-b7b2-7f8099397399","Type":"ContainerStarted","Data":"5c77c41a165650c8b39886171cc2240276dc23479d4da10496fb163f00afda88"} Jul 10 19:08:49.100106 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.099709 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xxz79" podStartSLOduration=66.795082297 podStartE2EDuration="1m8.099690428s" totalImagesPullingTime="1.304608131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:49.099619333 +0000 UTC m=+69.138492377" watchObservedRunningTime="2026-07-10 19:08:49.099690428 +0000 UTC m=+69.138563446" Jul 10 19:08:49.181370 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.181294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:49.235470 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.235433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:49.636079 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.636015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:49.678757 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.678721 2579 patch_prober.go:28] interesting pod/image-registry-666d58b98c-zz945 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 19:08:49.678908 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.678783 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-666d58b98c-zz945" podUID="40aec87c-18c7-4d15-af90-4f86885b6464" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:08:49.735679 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:49.735617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:51.235476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:51.235438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:51.332436 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:51.332396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:51.433498 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:51.433458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:54.625603 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.625568 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78d7b9c4dd-mpp98"] Jul 10 19:08:54.628017 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.627993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.637723 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.637701 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 19:08:54.639297 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.639276 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d7b9c4dd-mpp98"] Jul 10 19:08:54.696368 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.696335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-trusted-ca-bundle\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.696515 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.696397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-oauth-serving-cert\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.696515 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.696432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-oauth-config\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.696515 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.696466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwhz\" (UniqueName: \"kubernetes.io/projected/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-kube-api-access-jgwhz\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.696515 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.696494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-serving-cert\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.696753 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.696516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-service-ca\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.696753 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.696543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-config\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.797922 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.797882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-oauth-serving-cert\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.797928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-oauth-config\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.797968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwhz\" (UniqueName: \"kubernetes.io/projected/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-kube-api-access-jgwhz\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.797999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-serving-cert\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.798023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-service-ca\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798372 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.798147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-config\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798372 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.798215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-trusted-ca-bundle\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798766 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.798742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-oauth-serving-cert\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.798876 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.798742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-service-ca\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.799122 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.799094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-config\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.799232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.799183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-trusted-ca-bundle\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.800793 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.800773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-oauth-config\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.800891 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.800801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-serving-cert\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.807455 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.807405 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwhz\" (UniqueName: \"kubernetes.io/projected/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-kube-api-access-jgwhz\") pod \"console-78d7b9c4dd-mpp98\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:54.834051 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.834025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:54.891514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.891446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:54.939672 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:54.939636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:08:57.132797 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:57.132722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:57.234301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:57.234263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:57.285248 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:57.285209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:59.674831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:08:59.674799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:09:00.652222 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:00.652189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:00.752843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:00.752811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:01.085927 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.085854 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d7b9c4dd-mpp98"] Jul 10 19:09:01.086867 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:09:01.086745 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830feb7e_bb0d_42ba_a1fc_121b25d73ddf.slice/crio-b6304940d46eff4a9b4e432dd0e366da6851b2d232d0bf15fbeca74291b1d382 WatchSource:0}: Error finding container b6304940d46eff4a9b4e432dd0e366da6851b2d232d0bf15fbeca74291b1d382: Status 404 returned error can't find the container with id b6304940d46eff4a9b4e432dd0e366da6851b2d232d0bf15fbeca74291b1d382 Jul 10 19:09:01.126596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.126574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7b9c4dd-mpp98" event={"ID":"830feb7e-bb0d-42ba-a1fc-121b25d73ddf","Type":"ContainerStarted","Data":"b6304940d46eff4a9b4e432dd0e366da6851b2d232d0bf15fbeca74291b1d382"} Jul 10 19:09:01.128808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.128788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zkfcv" event={"ID":"8f760c7e-ca57-4789-a78b-327fdec3f6b3","Type":"ContainerStarted","Data":"c9ebb6ae0757c010a38a4281d7f228e726755f554efd3dd37b39ecdb7c08576b"} Jul 10 19:09:01.129050 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.129034 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:09:01.130703 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.130682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cfcd9c6-wm2g6" event={"ID":"17c1563f-724f-4b56-becd-27544a07e426","Type":"ContainerStarted","Data":"0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6"} Jul 10 19:09:01.132364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.132319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" event={"ID":"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a","Type":"ContainerStarted","Data":"8b7c266e93c0e73469644cad303d35a1beacdc6e84c066d746fd1aa0d154cc5a"} Jul 10 19:09:01.145529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.145475 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zkfcv" podStartSLOduration=66.128467571 podStartE2EDuration="1m20.145461678s" totalImagesPullingTime="14.016994107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:01.144795896 +0000 UTC m=+81.183668915" watchObservedRunningTime="2026-07-10 19:09:01.145461678 +0000 UTC m=+81.184334695" Jul 10 19:09:01.166221 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.166174 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-575cfcd9c6-wm2g6" podStartSLOduration=0.723240292 podStartE2EDuration="15.166155107s" totalImagesPullingTime="14.442914815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:01.165512182 +0000 UTC m=+81.204385216" watchObservedRunningTime="2026-07-10 19:09:01.166155107 +0000 UTC m=+81.205028125" Jul 10 19:09:01.312319 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:01.312292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:02.138894 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.138754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ssw5f" event={"ID":"d1acace1-4b9b-402d-a6ab-f8d8f3a401cc","Type":"ContainerStarted","Data":"dc434f16f84c8a8d8db65c48cd9df31ca6561fa756aead8e45a4499055633cae"} Jul 10 19:09:02.140846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.140812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7b9c4dd-mpp98" event={"ID":"830feb7e-bb0d-42ba-a1fc-121b25d73ddf","Type":"ContainerStarted","Data":"1065eb43f7c6f21040cbda547add34b3de3a4279b094a3b9cd17770a19dd1009"} Jul 10 19:09:02.142632 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.142598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-fh6rn" event={"ID":"a7b4d22f-9d66-4edc-b47d-e60671ac5979","Type":"ContainerStarted","Data":"c9e45e14e99a54698cff90da6d8eb9e5362ec8d8be903bcb335734f193fda3fa"} Jul 10 19:09:02.142821 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.142799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-fh6rn" Jul 10 19:09:02.145267 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.144877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" event={"ID":"8fbf6a90-cbaf-41a7-abc3-de3f54620f0a","Type":"ContainerStarted","Data":"494f863abc73e9e1a2c8b674a4bbffa8065e5cb20d9e3e16e7a49f69b930a3a8"} Jul 10 19:09:02.150021 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.149998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-fh6rn" Jul 10 19:09:02.156943 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.156893 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ssw5f" podStartSLOduration=66.844312831 podStartE2EDuration="1m21.156873784s" totalImagesPullingTime="14.312560953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:02.155916613 +0000 UTC m=+82.194789631" watchObservedRunningTime="2026-07-10 19:09:02.156873784 +0000 UTC m=+82.195746874" Jul 10 19:09:02.173546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.173493 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-fh6rn" podStartSLOduration=3.214526462 podStartE2EDuration="23.173475029s" totalImagesPullingTime="19.958948567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:02.171235825 +0000 UTC m=+82.210108836" watchObservedRunningTime="2026-07-10 19:09:02.173475029 +0000 UTC m=+82.212348047" Jul 10 19:09:02.190200 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.190141 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-tnkss" podStartSLOduration=3.162021116 podStartE2EDuration="16.190123481s" totalImagesPullingTime="13.028102365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:02.189165874 +0000 UTC m=+82.228038891" watchObservedRunningTime="2026-07-10 19:09:02.190123481 +0000 UTC m=+82.228996498" Jul 10 19:09:02.210066 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:02.210016 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78d7b9c4dd-mpp98" podStartSLOduration=8.209969832 podStartE2EDuration="8.209969832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:02.209072763 +0000 UTC m=+82.247945804" watchObservedRunningTime="2026-07-10 19:09:02.209969832 +0000 UTC m=+82.248842849" Jul 10 19:09:03.557109 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.557058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:03.769365 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.769337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:03.827877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.827805 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6p2dl"] Jul 10 19:09:03.856416 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.856390 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.859295 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.859268 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 19:09:03.859435 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.859300 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 19:09:03.859435 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.859267 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 19:09:03.859435 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.859337 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cn7z6\"" Jul 10 19:09:03.862355 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.862329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:03.932264 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.932228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:03.978625 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b58e1945-6536-410c-bb40-90e374750ee1-metrics-client-ca\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.978800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-accelerators-collector-config\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.978800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-wtmp\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.978800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-tls\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.978947 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-root\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.978947 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-textfile\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.978947 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxfg\" (UniqueName: \"kubernetes.io/projected/b58e1945-6536-410c-bb40-90e374750ee1-kube-api-access-xkxfg\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.978947 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.978926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:03.979116 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:03.979009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-sys\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.032436 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.032410 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75fb498fbd-hgkfj" Jul 10 19:09:04.079497 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-wtmp\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.079497 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-tls\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.079713 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-root\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.079713 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-textfile\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.079713 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxfg\" (UniqueName: \"kubernetes.io/projected/b58e1945-6536-410c-bb40-90e374750ee1-kube-api-access-xkxfg\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.079713 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.079713 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-wtmp\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.079713 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-sys\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.080021 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b58e1945-6536-410c-bb40-90e374750ee1-metrics-client-ca\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.080021 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.079780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-accelerators-collector-config\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.080199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.080174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-textfile\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.080430 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.080410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-root\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.080538 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.080518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b58e1945-6536-410c-bb40-90e374750ee1-sys\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.081115 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.081088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-accelerators-collector-config\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.081251 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.081221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b58e1945-6536-410c-bb40-90e374750ee1-metrics-client-ca\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.083418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.083392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.083906 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.083881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b58e1945-6536-410c-bb40-90e374750ee1-node-exporter-tls\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.093177 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.092956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.096773 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.095622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxfg\" (UniqueName: \"kubernetes.io/projected/b58e1945-6536-410c-bb40-90e374750ee1-kube-api-access-xkxfg\") pod \"node-exporter-6p2dl\" (UID: \"b58e1945-6536-410c-bb40-90e374750ee1\") " pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.167552 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.167519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6p2dl" Jul 10 19:09:04.182361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.182331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.208593 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:09:04.208556 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58e1945_6536_410c_bb40_90e374750ee1.slice/crio-519f0e678e140ce21572e3022e825b171ad64518b400073d3df37f8dce9704b7 WatchSource:0}: Error finding container 519f0e678e140ce21572e3022e825b171ad64518b400073d3df37f8dce9704b7: Status 404 returned error can't find the container with id 519f0e678e140ce21572e3022e825b171ad64518b400073d3df37f8dce9704b7 Jul 10 19:09:04.283157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.283124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.384399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.384372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.447666 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.447635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.542368 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.542338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.691504 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.691429 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-666d58b98c-zz945" podUID="40aec87c-18c7-4d15-af90-4f86885b6464" containerName="registry" containerID="cri-o://1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b" gracePeriod=30 Jul 10 19:09:04.787347 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.787316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.854620 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.854587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:04.872781 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.872719 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:09:04.888435 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.888409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:09:04.888706 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.888689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892400 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892440 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892482 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892440 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892617 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892725 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892852 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 19:09:04.892949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.892919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ptmpr\"" Jul 10 19:09:04.904069 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.899777 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 19:09:04.939914 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.939890 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:09:04.940398 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.940373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:09:04.947055 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.946999 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:09:04.988943 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.988908 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:09:04.989372 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989472 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4ph\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-kube-api-access-tx4ph\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-config-out\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989719 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989719 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-web-config\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989719 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989870 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989870 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989972 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-config-volume\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.989972 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:04.990077 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:04.989996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.090973 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.090880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-installation-pull-secrets\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.090973 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.090926 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-registry-certificates\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.090973 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.090951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-image-registry-private-configuration\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.091199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.090995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-trusted-ca\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.091199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091049 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-696qg\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-kube-api-access-696qg\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.091199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091095 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.091199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091122 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40aec87c-18c7-4d15-af90-4f86885b6464-ca-trust-extracted\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.091199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091150 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-bound-sa-token\") pod \"40aec87c-18c7-4d15-af90-4f86885b6464\" (UID: \"40aec87c-18c7-4d15-af90-4f86885b6464\") " Jul 10 19:09:05.091440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-web-config\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-config-volume\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4ph\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-kube-api-access-tx4ph\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-config-out\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.091768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.091699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.093151 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.093124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.094168 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.093828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.094599 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:09:05.094575 2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 10 19:09:05.094722 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:09:05.094673 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls podName:62f50804-74ce-4fd5-86fd-292270a0f51b nodeName:}" failed. No retries permitted until 2026-07-10 19:09:05.594636256 +0000 UTC m=+85.633509254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b") : secret "alertmanager-main-tls" not found Jul 10 19:09:05.096159 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.096092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.096159 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.096096 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:05.099194 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.099110 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:05.101143 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.100855 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:09:05.101807 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.101778 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:05.104580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.104458 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-kube-api-access-696qg" (OuterVolumeSpecName: "kube-api-access-696qg") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "kube-api-access-696qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:09:05.104580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.104555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.105630 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.105607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-web-config\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.106996 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.106963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:09:05.110005 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.109152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40aec87c-18c7-4d15-af90-4f86885b6464-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:09:05.110005 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.109541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4ph\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-kube-api-access-tx4ph\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.112578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.112051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-config-out\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.112578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.112323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.112578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.112434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.112578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.112498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.112578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.112541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-config-volume\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.113040 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.112961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.114112 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.114083 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "40aec87c-18c7-4d15-af90-4f86885b6464" (UID: "40aec87c-18c7-4d15-af90-4f86885b6464"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:05.160236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.160206 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6p2dl" event={"ID":"b58e1945-6536-410c-bb40-90e374750ee1","Type":"ContainerStarted","Data":"519f0e678e140ce21572e3022e825b171ad64518b400073d3df37f8dce9704b7"} Jul 10 19:09:05.161950 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.161922 2579 generic.go:356] "Generic (PLEG): container finished" podID="40aec87c-18c7-4d15-af90-4f86885b6464" containerID="1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b" exitCode=0 Jul 10 19:09:05.162323 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.162298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d58b98c-zz945" Jul 10 19:09:05.162323 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.162312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d58b98c-zz945" event={"ID":"40aec87c-18c7-4d15-af90-4f86885b6464","Type":"ContainerDied","Data":"1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b"} Jul 10 19:09:05.162462 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.162354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d58b98c-zz945" event={"ID":"40aec87c-18c7-4d15-af90-4f86885b6464","Type":"ContainerDied","Data":"2cbffb0fa3f32c2db96bcfa3fde5353250aa15f4b8e816a74785e044cf11b83c"} Jul 10 19:09:05.162462 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.162379 2579 scope.go:117] "RemoveContainer" containerID="1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b" Jul 10 19:09:05.168156 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.168099 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:09:05.174915 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.174847 2579 scope.go:117] "RemoveContainer" containerID="1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b" Jul 10 19:09:05.175167 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:09:05.175146 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b\": container with ID starting with 1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b not found: ID does not exist" containerID="1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b" Jul 10 19:09:05.175245 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.175175 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b"} err="failed to get container status \"1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b\": rpc error: code = NotFound desc = could not find container \"1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b\": container with ID starting with 1b56b06e6c71aea0d63db63493d5c0139d92b7bd26bc2578376d1d82e974a99b not found: ID does not exist" Jul 10 19:09:05.191487 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.191465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:05.192703 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192679 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696qg\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-kube-api-access-696qg\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.192703 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192702 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-registry-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.192850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192718 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40aec87c-18c7-4d15-af90-4f86885b6464-ca-trust-extracted\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.192850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192731 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40aec87c-18c7-4d15-af90-4f86885b6464-bound-sa-token\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.192850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192744 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-installation-pull-secrets\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.192850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192757 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-registry-certificates\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.192850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192771 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40aec87c-18c7-4d15-af90-4f86885b6464-image-registry-private-configuration\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.192850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.192785 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40aec87c-18c7-4d15-af90-4f86885b6464-trusted-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:05.204265 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.204213 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666d58b98c-zz945"] Jul 10 19:09:05.213714 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.213217 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-666d58b98c-zz945"] Jul 10 19:09:05.218427 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.218407 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575cfcd9c6-wm2g6"] Jul 10 19:09:05.596894 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.596852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.599670 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.599626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.803788 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.803755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:09:05.947337 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:05.947282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:06.008643 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.008160 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:09:06.008782 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.008601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:06.009977 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:09:06.009950 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f50804_74ce_4fd5_86fd_292270a0f51b.slice/crio-08371d93002d935e4948f8055454660cb6842f32621ad605a0a9063670f0986e WatchSource:0}: Error finding container 08371d93002d935e4948f8055454660cb6842f32621ad605a0a9063670f0986e: Status 404 returned error can't find the container with id 08371d93002d935e4948f8055454660cb6842f32621ad605a0a9063670f0986e Jul 10 19:09:06.113054 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.112997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:06.169424 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.169387 2579 generic.go:356] "Generic (PLEG): container finished" podID="b58e1945-6536-410c-bb40-90e374750ee1" containerID="5c41c9587af52205f598e35c9dc20af024e8bef6253a18123e397932e6673acc" exitCode=0 Jul 10 19:09:06.169577 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.169479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6p2dl" event={"ID":"b58e1945-6536-410c-bb40-90e374750ee1","Type":"ContainerDied","Data":"5c41c9587af52205f598e35c9dc20af024e8bef6253a18123e397932e6673acc"} Jul 10 19:09:06.171005 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.170974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerStarted","Data":"08371d93002d935e4948f8055454660cb6842f32621ad605a0a9063670f0986e"} Jul 10 19:09:06.191999 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.191977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:06.357642 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.357609 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:09:06.694374 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.694276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:06.793618 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.793583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:06.801320 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.801289 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76c5dfb559-fg88n"] Jul 10 19:09:06.801746 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.801719 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40aec87c-18c7-4d15-af90-4f86885b6464" containerName="registry" Jul 10 19:09:06.801746 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.801742 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40aec87c-18c7-4d15-af90-4f86885b6464" containerName="registry" Jul 10 19:09:06.802105 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.801807 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40aec87c-18c7-4d15-af90-4f86885b6464" containerName="registry" Jul 10 19:09:06.816577 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.816549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76c5dfb559-fg88n"] Jul 10 19:09:06.816987 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.816723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.820356 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.819771 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 19:09:06.820356 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.820013 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 19:09:06.820356 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.820105 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rpt46\"" Jul 10 19:09:06.820356 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.820206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 19:09:06.820634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.820418 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9sqjm4c1icnme\"" Jul 10 19:09:06.820634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.820422 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 19:09:06.820816 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.820793 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 19:09:06.874758 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.874728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqffv\" (UniqueName: \"kubernetes.io/projected/0ab4184a-cba3-448c-b2cc-422618f2a863-kube-api-access-wqffv\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-tls\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab4184a-cba3-448c-b2cc-422618f2a863-metrics-client-ca\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:06.912311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:06.911778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-grpc-tls\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.012849 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.012763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab4184a-cba3-448c-b2cc-422618f2a863-metrics-client-ca\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.012849 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.012810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-grpc-tls\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.013325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.012852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqffv\" (UniqueName: \"kubernetes.io/projected/0ab4184a-cba3-448c-b2cc-422618f2a863-kube-api-access-wqffv\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.013325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.012893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.013325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.012930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.013325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.012984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.013325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.013012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-tls\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.013325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.013042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.014206 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.014140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab4184a-cba3-448c-b2cc-422618f2a863-metrics-client-ca\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.016757 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.016531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-grpc-tls\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.016757 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.016982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.018634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.018014 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.018634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.018595 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.019365 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.019056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.020197 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.020140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:07.020368 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.020347 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ab4184a-cba3-448c-b2cc-422618f2a863-secret-thanos-querier-tls\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.022959 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.022935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqffv\" (UniqueName: \"kubernetes.io/projected/0ab4184a-cba3-448c-b2cc-422618f2a863-kube-api-access-wqffv\") pod \"thanos-querier-76c5dfb559-fg88n\" (UID: \"0ab4184a-cba3-448c-b2cc-422618f2a863\") " pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.070800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.070773 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40aec87c-18c7-4d15-af90-4f86885b6464" path="/var/lib/kubelet/pods/40aec87c-18c7-4d15-af90-4f86885b6464/volumes" Jul 10 19:09:07.130012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.129956 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:07.178905 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.178854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6p2dl" event={"ID":"b58e1945-6536-410c-bb40-90e374750ee1","Type":"ContainerStarted","Data":"a30e65325d420c3c220ac2d2bae8bb961766fef90810ce1ce534e8770b985c24"} Jul 10 19:09:07.179057 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.178972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6p2dl" event={"ID":"b58e1945-6536-410c-bb40-90e374750ee1","Type":"ContainerStarted","Data":"cc97f9bccb6ff41f34202ee951f9a562b03101259c0304f9d0ccb18db0dc5bab"} Jul 10 19:09:07.198038 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.197840 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6p2dl" podStartSLOduration=2.462192796 podStartE2EDuration="4.197823733s" totalImagesPullingTime="1.643197352s" totalInitContainerRuntime="92.433585ms" isStatefulPod=true podCreationTimestamp="2026-07-10 19:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:07.197042319 +0000 UTC m=+87.235915337" watchObservedRunningTime="2026-07-10 19:09:07.197823733 +0000 UTC m=+87.236696751" Jul 10 19:09:07.384168 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.384134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:07.484625 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.484594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:07.608865 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:07.608833 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76c5dfb559-fg88n"] Jul 10 19:09:07.617547 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:09:07.617514 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab4184a_cba3_448c_b2cc_422618f2a863.slice/crio-e14c14c1e1ae235eb8606f1da41ea6780594f794d655c9aaa8755f8bf5dff352 WatchSource:0}: Error finding container e14c14c1e1ae235eb8606f1da41ea6780594f794d655c9aaa8755f8bf5dff352: Status 404 returned error can't find the container with id e14c14c1e1ae235eb8606f1da41ea6780594f794d655c9aaa8755f8bf5dff352 Jul 10 19:09:08.171216 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.171138 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5cff96cdbf-rvvvp"] Jul 10 19:09:08.189270 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.188464 2579 generic.go:356] "Generic (PLEG): container finished" podID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerID="ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0" exitCode=0 Jul 10 19:09:08.192044 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.191351 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cff96cdbf-rvvvp"] Jul 10 19:09:08.192044 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.191392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0"} Jul 10 19:09:08.192044 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.191447 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" event={"ID":"0ab4184a-cba3-448c-b2cc-422618f2a863","Type":"ContainerStarted","Data":"e14c14c1e1ae235eb8606f1da41ea6780594f794d655c9aaa8755f8bf5dff352"} Jul 10 19:09:08.192044 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.191569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.199831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.195274 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 19:09:08.199831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.195797 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 19:09:08.199831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.195977 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 19:09:08.199831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.195977 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 19:09:08.199831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.196190 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gwdq5\"" Jul 10 19:09:08.199831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.196821 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-51lfnkkdg4huk\"" Jul 10 19:09:08.326468 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.326434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-client-ca-bundle\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.326918 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.326712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/767bee55-2625-4e10-a3b5-867390c1ee06-audit-log\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.327175 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.327086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767bee55-2625-4e10-a3b5-867390c1ee06-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.327299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.327227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-secret-metrics-server-tls\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.327635 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.327467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-secret-metrics-server-client-certs\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.327772 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.327678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/767bee55-2625-4e10-a3b5-867390c1ee06-metrics-server-audit-profiles\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.327910 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.327879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6kx2\" (UniqueName: \"kubernetes.io/projected/767bee55-2625-4e10-a3b5-867390c1ee06-kube-api-access-f6kx2\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429150 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/767bee55-2625-4e10-a3b5-867390c1ee06-metrics-server-audit-profiles\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429150 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6kx2\" (UniqueName: \"kubernetes.io/projected/767bee55-2625-4e10-a3b5-867390c1ee06-kube-api-access-f6kx2\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429150 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-client-ca-bundle\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/767bee55-2625-4e10-a3b5-867390c1ee06-audit-log\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767bee55-2625-4e10-a3b5-867390c1ee06-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-secret-metrics-server-tls\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-secret-metrics-server-client-certs\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.429958 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.429923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/767bee55-2625-4e10-a3b5-867390c1ee06-audit-log\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.430191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.430136 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/767bee55-2625-4e10-a3b5-867390c1ee06-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.430277 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.430252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/767bee55-2625-4e10-a3b5-867390c1ee06-metrics-server-audit-profiles\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.432115 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.432089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-secret-metrics-server-client-certs\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.432322 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.432299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-secret-metrics-server-tls\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.432697 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.432671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/767bee55-2625-4e10-a3b5-867390c1ee06-client-ca-bundle\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.437760 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.437737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6kx2\" (UniqueName: \"kubernetes.io/projected/767bee55-2625-4e10-a3b5-867390c1ee06-kube-api-access-f6kx2\") pod \"metrics-server-5cff96cdbf-rvvvp\" (UID: \"767bee55-2625-4e10-a3b5-867390c1ee06\") " pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.505798 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.505771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:08.585138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.585107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:08.684185 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.683976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:08.717210 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:08.717139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cff96cdbf-rvvvp"] Jul 10 19:09:09.072914 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:09.072767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:09.197197 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:09.197152 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" event={"ID":"767bee55-2625-4e10-a3b5-867390c1ee06","Type":"ContainerStarted","Data":"03da9078f4bfe25b297b38f2c8bd84802ff8596d174d2e2dcb0b669c48ce6458"} Jul 10 19:09:09.473968 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:09.473898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:10.142871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.142836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:10.151269 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.150724 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:09:10.170108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.170080 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:09:10.170258 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.170194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.172853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.172827 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 19:09:10.173725 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.173528 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 19:09:10.173725 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.173556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 19:09:10.173891 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.173769 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qlfr5\"" Jul 10 19:09:10.173953 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.173892 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 19:09:10.174010 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.173986 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 19:09:10.174130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.174093 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 19:09:10.174362 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.174237 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 19:09:10.174460 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.174360 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 19:09:10.175382 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.175264 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 19:09:10.176700 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.175722 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 19:09:10.193260 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.190064 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 19:09:10.193260 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.191776 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 19:09:10.193260 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.192963 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-atsv3quggohf\"" Jul 10 19:09:10.201218 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.201198 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 19:09:10.204249 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.204220 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 19:09:10.249018 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.248980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-config\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249182 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nslx\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-kube-api-access-6nslx\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.249506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.250018 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.250018 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.250018 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.249580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.350813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.350763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.350980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.350822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.350980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.350865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.350980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.350886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.350980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.350910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-config\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.350980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.350932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351542 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351513 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351730 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nslx\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-kube-api-access-6nslx\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.351938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.351978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.352034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.352063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.352127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352379 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.352164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352379 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.352209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.352379 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.352239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.355412 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.354413 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-config\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.355412 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.355065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.355862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.356895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.357869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.358562 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.359123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359572 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.359490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359635 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.359593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359739 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.359694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.359868 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.359805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.360263 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.360240 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.360428 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.360387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.360928 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.360886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.361012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.361001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.361381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.361351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.362357 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.362338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.364822 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.364770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nslx\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-kube-api-access-6nslx\") pod \"prometheus-k8s-0\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:10.503067 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:10.502940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:12.094755 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:12.094710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:09:12.095878 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:09:12.095837 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bad7f1c_26e6_4232_a047_ad133bd255cf.slice/crio-24a9e88cad51b65f2a18b64f8891ab23e6e760f9cf8ad67e76983773368ead48 WatchSource:0}: Error finding container 24a9e88cad51b65f2a18b64f8891ab23e6e760f9cf8ad67e76983773368ead48: Status 404 returned error can't find the container with id 24a9e88cad51b65f2a18b64f8891ab23e6e760f9cf8ad67e76983773368ead48 Jul 10 19:09:12.214086 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:12.213990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb"} Jul 10 19:09:12.214257 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:12.214107 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"24a9e88cad51b65f2a18b64f8891ab23e6e760f9cf8ad67e76983773368ead48"} Jul 10 19:09:12.218106 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:12.218079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerStarted","Data":"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5"} Jul 10 19:09:12.220005 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:12.219977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" event={"ID":"767bee55-2625-4e10-a3b5-867390c1ee06","Type":"ContainerStarted","Data":"6ccce06b379a5efc79d418485404757da55bf22bcd9b53bdd640c71f198bb95e"} Jul 10 19:09:12.221502 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:12.221477 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" event={"ID":"0ab4184a-cba3-448c-b2cc-422618f2a863","Type":"ContainerStarted","Data":"26d0f8e52feb5766c79bef35840df2734d43a0f9e28feefa474c40f064c204f5"} Jul 10 19:09:12.262503 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:12.262451 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" podStartSLOduration=1.038228398 podStartE2EDuration="4.262431603s" totalImagesPullingTime="3.224203205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:12.261702083 +0000 UTC m=+92.300575124" watchObservedRunningTime="2026-07-10 19:09:12.262431603 +0000 UTC m=+92.301304621" Jul 10 19:09:13.228070 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.228033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" event={"ID":"0ab4184a-cba3-448c-b2cc-422618f2a863","Type":"ContainerStarted","Data":"4458fa4e6dcaf09ff70d0411817d0e5710de6072dae8d8c071ec9c913919d9d8"} Jul 10 19:09:13.228541 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.228122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" event={"ID":"0ab4184a-cba3-448c-b2cc-422618f2a863","Type":"ContainerStarted","Data":"074b937597b0b8f0d3a12561c8ad202da10e0e1a141d1c40c5033d01ad4f3ed3"} Jul 10 19:09:13.229423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.229400 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerID="57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb" exitCode=0 Jul 10 19:09:13.229519 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.229477 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb"} Jul 10 19:09:13.232567 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.232546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerStarted","Data":"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823"} Jul 10 19:09:13.232686 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.232598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerStarted","Data":"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea"} Jul 10 19:09:13.232686 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.232627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerStarted","Data":"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949"} Jul 10 19:09:13.232686 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:13.232670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerStarted","Data":"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc"} Jul 10 19:09:14.240604 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:14.240567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerStarted","Data":"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3"} Jul 10 19:09:14.244108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:14.244076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" event={"ID":"0ab4184a-cba3-448c-b2cc-422618f2a863","Type":"ContainerStarted","Data":"65ffb86cdca6831970824f97123e9ba423308c3577149d0cd9f0607752992005"} Jul 10 19:09:14.244237 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:14.244153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" event={"ID":"0ab4184a-cba3-448c-b2cc-422618f2a863","Type":"ContainerStarted","Data":"517b6c15825dd0f68a9e537f06d705af840046215dac7e1fd691c38279f95493"} Jul 10 19:09:14.244237 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:14.244197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" event={"ID":"0ab4184a-cba3-448c-b2cc-422618f2a863","Type":"ContainerStarted","Data":"916ae48bcac7e9b5e1b8705858eba284d418658ea5decc30468c295b65f570a6"} Jul 10 19:09:14.244341 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:14.244244 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:14.271297 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:14.271255 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.456318164 podStartE2EDuration="10.271240986s" totalImagesPullingTime="6.801550731s" totalInitContainerRuntime="13.372091ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:14.269323937 +0000 UTC m=+94.308196958" watchObservedRunningTime="2026-07-10 19:09:14.271240986 +0000 UTC m=+94.310114002" Jul 10 19:09:14.300067 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:14.300003 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" podStartSLOduration=2.744151082 podStartE2EDuration="8.299983579s" totalImagesPullingTime="5.555832497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:14.298108752 +0000 UTC m=+94.336981798" watchObservedRunningTime="2026-07-10 19:09:14.299983579 +0000 UTC m=+94.338856597" Jul 10 19:09:15.076712 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:15.076670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:15.176383 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:15.176350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:15.240339 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:15.240306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:18.267249 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:18.267179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3"} Jul 10 19:09:18.267249 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:18.267242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821"} Jul 10 19:09:18.267679 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:18.267274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b"} Jul 10 19:09:18.267679 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:18.267303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0"} Jul 10 19:09:18.267679 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:18.267328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf"} Jul 10 19:09:18.267679 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:18.267366 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerStarted","Data":"4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c"} Jul 10 19:09:18.298100 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:18.298054 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.151064638 podStartE2EDuration="8.298039547s" totalImagesPullingTime="4.129701227s" totalInitContainerRuntime="17.273682ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:18.295478929 +0000 UTC m=+98.334351945" watchObservedRunningTime="2026-07-10 19:09:18.298039547 +0000 UTC m=+98.336912564" Jul 10 19:09:20.141384 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:20.141346 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d7b9c4dd-mpp98"] Jul 10 19:09:20.244244 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:20.244209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:20.256369 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:20.256345 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76c5dfb559-fg88n" Jul 10 19:09:20.345131 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:20.345102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:20.444790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:20.444703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:20.503794 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:20.503773 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:09:21.698364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:21.698327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:21.799003 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:21.798976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:28.506270 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:28.506237 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:28.506844 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:28.506292 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:30.243753 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.243695 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-575cfcd9c6-wm2g6" podUID="17c1563f-724f-4b56-becd-27544a07e426" containerName="console" containerID="cri-o://0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6" gracePeriod=15 Jul 10 19:09:30.505790 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.505736 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:09:30.618867 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.618843 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-service-ca\") pod \"17c1563f-724f-4b56-becd-27544a07e426\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " Jul 10 19:09:30.619002 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.618890 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-serving-cert\") pod \"17c1563f-724f-4b56-becd-27544a07e426\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " Jul 10 19:09:30.619002 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.618911 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hltsr\" (UniqueName: \"kubernetes.io/projected/17c1563f-724f-4b56-becd-27544a07e426-kube-api-access-hltsr\") pod \"17c1563f-724f-4b56-becd-27544a07e426\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " Jul 10 19:09:30.619082 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619033 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-console-config\") pod \"17c1563f-724f-4b56-becd-27544a07e426\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " Jul 10 19:09:30.619130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619113 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-oauth-config\") pod \"17c1563f-724f-4b56-becd-27544a07e426\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " Jul 10 19:09:30.619287 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619256 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-oauth-serving-cert\") pod \"17c1563f-724f-4b56-becd-27544a07e426\" (UID: \"17c1563f-724f-4b56-becd-27544a07e426\") " Jul 10 19:09:30.619381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-service-ca" (OuterVolumeSpecName: "service-ca") pod "17c1563f-724f-4b56-becd-27544a07e426" (UID: "17c1563f-724f-4b56-becd-27544a07e426"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:30.619450 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619409 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-console-config" (OuterVolumeSpecName: "console-config") pod "17c1563f-724f-4b56-becd-27544a07e426" (UID: "17c1563f-724f-4b56-becd-27544a07e426"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:30.619636 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619606 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17c1563f-724f-4b56-becd-27544a07e426" (UID: "17c1563f-724f-4b56-becd-27544a07e426"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:30.619734 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619634 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-console-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:30.619734 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.619676 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:30.621206 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.621184 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c1563f-724f-4b56-becd-27544a07e426-kube-api-access-hltsr" (OuterVolumeSpecName: "kube-api-access-hltsr") pod "17c1563f-724f-4b56-becd-27544a07e426" (UID: "17c1563f-724f-4b56-becd-27544a07e426"). InnerVolumeSpecName "kube-api-access-hltsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:09:30.621283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.621216 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17c1563f-724f-4b56-becd-27544a07e426" (UID: "17c1563f-724f-4b56-becd-27544a07e426"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:30.621283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.621269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17c1563f-724f-4b56-becd-27544a07e426" (UID: "17c1563f-724f-4b56-becd-27544a07e426"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:30.720701 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.720680 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17c1563f-724f-4b56-becd-27544a07e426-oauth-serving-cert\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:30.720701 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.720699 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-serving-cert\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:30.720819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.720708 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hltsr\" (UniqueName: \"kubernetes.io/projected/17c1563f-724f-4b56-becd-27544a07e426-kube-api-access-hltsr\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:30.720819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:30.720718 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17c1563f-724f-4b56-becd-27544a07e426-console-oauth-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:31.319985 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.319953 2579 generic.go:356] "Generic (PLEG): container finished" podID="17c1563f-724f-4b56-becd-27544a07e426" containerID="0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6" exitCode=0 Jul 10 19:09:31.320526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.319997 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cfcd9c6-wm2g6" Jul 10 19:09:31.320526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.320020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cfcd9c6-wm2g6" event={"ID":"17c1563f-724f-4b56-becd-27544a07e426","Type":"ContainerDied","Data":"0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6"} Jul 10 19:09:31.320526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.320062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cfcd9c6-wm2g6" event={"ID":"17c1563f-724f-4b56-becd-27544a07e426","Type":"ContainerDied","Data":"1f44986066df6fac67de6c2af579409fda085d62601e30f254ad056fe69d9b68"} Jul 10 19:09:31.320526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.320084 2579 scope.go:117] "RemoveContainer" containerID="0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6" Jul 10 19:09:31.336732 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.336702 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575cfcd9c6-wm2g6"] Jul 10 19:09:31.340594 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.340575 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-575cfcd9c6-wm2g6"] Jul 10 19:09:31.344610 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.344592 2579 scope.go:117] "RemoveContainer" containerID="0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6" Jul 10 19:09:31.344882 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:09:31.344865 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6\": container with ID starting with 0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6 not found: ID does not exist" containerID="0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6" Jul 10 19:09:31.344945 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:31.344893 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6"} err="failed to get container status \"0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6\": rpc error: code = NotFound desc = could not find container \"0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6\": container with ID starting with 0ee8f87467cb91d46315cd4c4437f1f5a1bc2e995a2b5482d6f4341af5bbe7f6 not found: ID does not exist" Jul 10 19:09:32.147109 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:32.147084 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zkfcv" Jul 10 19:09:33.070558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:33.070522 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c1563f-724f-4b56-becd-27544a07e426" path="/var/lib/kubelet/pods/17c1563f-724f-4b56-becd-27544a07e426/volumes" Jul 10 19:09:36.665195 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:36.665154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:36.720462 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:36.720416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:45.161455 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.161411 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78d7b9c4dd-mpp98" podUID="830feb7e-bb0d-42ba-a1fc-121b25d73ddf" containerName="console" containerID="cri-o://1065eb43f7c6f21040cbda547add34b3de3a4279b094a3b9cd17770a19dd1009" gracePeriod=15 Jul 10 19:09:45.165929 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.165894 2579 patch_prober.go:28] interesting pod/console-78d7b9c4dd-mpp98 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.17:8443/health\": EOF" start-of-body= Jul 10 19:09:45.166062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.165986 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-78d7b9c4dd-mpp98" podUID="830feb7e-bb0d-42ba-a1fc-121b25d73ddf" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": EOF" Jul 10 19:09:45.380549 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.380520 2579 generic.go:356] "Generic (PLEG): container finished" podID="830feb7e-bb0d-42ba-a1fc-121b25d73ddf" containerID="1065eb43f7c6f21040cbda547add34b3de3a4279b094a3b9cd17770a19dd1009" exitCode=0 Jul 10 19:09:45.380715 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.380593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7b9c4dd-mpp98" event={"ID":"830feb7e-bb0d-42ba-a1fc-121b25d73ddf","Type":"ContainerDied","Data":"1065eb43f7c6f21040cbda547add34b3de3a4279b094a3b9cd17770a19dd1009"} Jul 10 19:09:45.437931 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.437879 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:09:45.537995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.537970 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-service-ca\") pod \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " Jul 10 19:09:45.538126 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538075 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-config\") pod \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " Jul 10 19:09:45.538189 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538145 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-oauth-serving-cert\") pod \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " Jul 10 19:09:45.538247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-trusted-ca-bundle\") pod \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " Jul 10 19:09:45.538247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538231 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-service-ca" (OuterVolumeSpecName: "service-ca") pod "830feb7e-bb0d-42ba-a1fc-121b25d73ddf" (UID: "830feb7e-bb0d-42ba-a1fc-121b25d73ddf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:45.538347 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538253 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-serving-cert\") pod \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " Jul 10 19:09:45.538347 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgwhz\" (UniqueName: \"kubernetes.io/projected/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-kube-api-access-jgwhz\") pod \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " Jul 10 19:09:45.538347 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538331 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-oauth-config\") pod \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\" (UID: \"830feb7e-bb0d-42ba-a1fc-121b25d73ddf\") " Jul 10 19:09:45.538488 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538401 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-config" (OuterVolumeSpecName: "console-config") pod "830feb7e-bb0d-42ba-a1fc-121b25d73ddf" (UID: "830feb7e-bb0d-42ba-a1fc-121b25d73ddf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:45.538587 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538562 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "830feb7e-bb0d-42ba-a1fc-121b25d73ddf" (UID: "830feb7e-bb0d-42ba-a1fc-121b25d73ddf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:45.538689 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538625 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:45.538689 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538681 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:45.538767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.538625 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "830feb7e-bb0d-42ba-a1fc-121b25d73ddf" (UID: "830feb7e-bb0d-42ba-a1fc-121b25d73ddf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:09:45.540541 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.540516 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "830feb7e-bb0d-42ba-a1fc-121b25d73ddf" (UID: "830feb7e-bb0d-42ba-a1fc-121b25d73ddf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:45.540631 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.540513 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-kube-api-access-jgwhz" (OuterVolumeSpecName: "kube-api-access-jgwhz") pod "830feb7e-bb0d-42ba-a1fc-121b25d73ddf" (UID: "830feb7e-bb0d-42ba-a1fc-121b25d73ddf"). InnerVolumeSpecName "kube-api-access-jgwhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:09:45.540631 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.540578 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "830feb7e-bb0d-42ba-a1fc-121b25d73ddf" (UID: "830feb7e-bb0d-42ba-a1fc-121b25d73ddf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:09:45.640141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.640041 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-serving-cert\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:45.640141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.640070 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwhz\" (UniqueName: \"kubernetes.io/projected/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-kube-api-access-jgwhz\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:45.640141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.640084 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-console-oauth-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:45.640141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.640095 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-oauth-serving-cert\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:45.640141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:45.640106 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/830feb7e-bb0d-42ba-a1fc-121b25d73ddf-trusted-ca-bundle\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:09:46.386172 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:46.386145 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7b9c4dd-mpp98" Jul 10 19:09:46.386535 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:46.386143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7b9c4dd-mpp98" event={"ID":"830feb7e-bb0d-42ba-a1fc-121b25d73ddf","Type":"ContainerDied","Data":"b6304940d46eff4a9b4e432dd0e366da6851b2d232d0bf15fbeca74291b1d382"} Jul 10 19:09:46.386535 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:46.386280 2579 scope.go:117] "RemoveContainer" containerID="1065eb43f7c6f21040cbda547add34b3de3a4279b094a3b9cd17770a19dd1009" Jul 10 19:09:46.406732 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:46.406708 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d7b9c4dd-mpp98"] Jul 10 19:09:46.408195 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:46.408173 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78d7b9c4dd-mpp98"] Jul 10 19:09:47.070701 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:47.070636 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="830feb7e-bb0d-42ba-a1fc-121b25d73ddf" path="/var/lib/kubelet/pods/830feb7e-bb0d-42ba-a1fc-121b25d73ddf/volumes" Jul 10 19:09:47.665370 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:47.665335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:47.718885 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:47.718847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:48.511129 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:48.511105 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:09:48.514949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:09:48.514925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5cff96cdbf-rvvvp" Jul 10 19:10:10.503245 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:10.503216 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:10.522172 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:10.522147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:11.499989 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:11.499959 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:23.123137 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:23.123102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:23.214059 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:23.214019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:24.171071 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.171036 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:10:24.171475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.171326 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="alertmanager" containerID="cri-o://6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5" gracePeriod=120 Jul 10 19:10:24.171475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.171344 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-web" containerID="cri-o://faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949" gracePeriod=120 Jul 10 19:10:24.171475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.171371 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-metric" containerID="cri-o://165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823" gracePeriod=120 Jul 10 19:10:24.171475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.171381 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="prom-label-proxy" containerID="cri-o://48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3" gracePeriod=120 Jul 10 19:10:24.171475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.171327 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy" containerID="cri-o://65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea" gracePeriod=120 Jul 10 19:10:24.171475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.171356 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="config-reloader" containerID="cri-o://170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc" gracePeriod=120 Jul 10 19:10:24.537529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537454 2579 generic.go:356] "Generic (PLEG): container finished" podID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerID="48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3" exitCode=0 Jul 10 19:10:24.537529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537478 2579 generic.go:356] "Generic (PLEG): container finished" podID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerID="65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea" exitCode=0 Jul 10 19:10:24.537529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537484 2579 generic.go:356] "Generic (PLEG): container finished" podID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerID="170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc" exitCode=0 Jul 10 19:10:24.537529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537491 2579 generic.go:356] "Generic (PLEG): container finished" podID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerID="6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5" exitCode=0 Jul 10 19:10:24.537529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3"} Jul 10 19:10:24.537791 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea"} Jul 10 19:10:24.537791 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc"} Jul 10 19:10:24.537791 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:24.537638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5"} Jul 10 19:10:25.510433 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.510408 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:25.546978 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.546946 2579 generic.go:356] "Generic (PLEG): container finished" podID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerID="165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823" exitCode=0 Jul 10 19:10:25.546978 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.546974 2579 generic.go:356] "Generic (PLEG): container finished" podID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerID="faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949" exitCode=0 Jul 10 19:10:25.547191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.547021 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823"} Jul 10 19:10:25.547191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.547040 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:25.547191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.547086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949"} Jul 10 19:10:25.547191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.547114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62f50804-74ce-4fd5-86fd-292270a0f51b","Type":"ContainerDied","Data":"08371d93002d935e4948f8055454660cb6842f32621ad605a0a9063670f0986e"} Jul 10 19:10:25.547191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.547144 2579 scope.go:117] "RemoveContainer" containerID="48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3" Jul 10 19:10:25.556110 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.556086 2579 scope.go:117] "RemoveContainer" containerID="165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823" Jul 10 19:10:25.563056 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.563029 2579 scope.go:117] "RemoveContainer" containerID="65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea" Jul 10 19:10:25.570227 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.570212 2579 scope.go:117] "RemoveContainer" containerID="faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949" Jul 10 19:10:25.577534 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.577511 2579 scope.go:117] "RemoveContainer" containerID="170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc" Jul 10 19:10:25.584341 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.584322 2579 scope.go:117] "RemoveContainer" containerID="6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5" Jul 10 19:10:25.593031 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.593016 2579 scope.go:117] "RemoveContainer" containerID="ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0" Jul 10 19:10:25.599578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.599560 2579 scope.go:117] "RemoveContainer" containerID="48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3" Jul 10 19:10:25.599860 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:25.599844 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3\": container with ID starting with 48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3 not found: ID does not exist" containerID="48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3" Jul 10 19:10:25.599913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.599869 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3"} err="failed to get container status \"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3\": rpc error: code = NotFound desc = could not find container \"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3\": container with ID starting with 48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3 not found: ID does not exist" Jul 10 19:10:25.599913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.599886 2579 scope.go:117] "RemoveContainer" containerID="165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823" Jul 10 19:10:25.600091 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:25.600077 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823\": container with ID starting with 165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823 not found: ID does not exist" containerID="165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823" Jul 10 19:10:25.600129 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600094 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823"} err="failed to get container status \"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823\": rpc error: code = NotFound desc = could not find container \"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823\": container with ID starting with 165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823 not found: ID does not exist" Jul 10 19:10:25.600129 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600106 2579 scope.go:117] "RemoveContainer" containerID="65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea" Jul 10 19:10:25.600331 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:25.600315 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea\": container with ID starting with 65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea not found: ID does not exist" containerID="65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea" Jul 10 19:10:25.600378 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600335 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea"} err="failed to get container status \"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea\": rpc error: code = NotFound desc = could not find container \"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea\": container with ID starting with 65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea not found: ID does not exist" Jul 10 19:10:25.600378 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600348 2579 scope.go:117] "RemoveContainer" containerID="faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949" Jul 10 19:10:25.600541 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:25.600525 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949\": container with ID starting with faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949 not found: ID does not exist" containerID="faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949" Jul 10 19:10:25.600579 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600545 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949"} err="failed to get container status \"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949\": rpc error: code = NotFound desc = could not find container \"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949\": container with ID starting with faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949 not found: ID does not exist" Jul 10 19:10:25.600579 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600559 2579 scope.go:117] "RemoveContainer" containerID="170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc" Jul 10 19:10:25.600765 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:25.600749 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc\": container with ID starting with 170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc not found: ID does not exist" containerID="170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc" Jul 10 19:10:25.600812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600768 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc"} err="failed to get container status \"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc\": rpc error: code = NotFound desc = could not find container \"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc\": container with ID starting with 170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc not found: ID does not exist" Jul 10 19:10:25.600812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600782 2579 scope.go:117] "RemoveContainer" containerID="6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5" Jul 10 19:10:25.600962 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:25.600947 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5\": container with ID starting with 6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5 not found: ID does not exist" containerID="6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5" Jul 10 19:10:25.600999 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600965 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5"} err="failed to get container status \"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5\": rpc error: code = NotFound desc = could not find container \"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5\": container with ID starting with 6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5 not found: ID does not exist" Jul 10 19:10:25.600999 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.600976 2579 scope.go:117] "RemoveContainer" containerID="ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0" Jul 10 19:10:25.601154 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:25.601139 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0\": container with ID starting with ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0 not found: ID does not exist" containerID="ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0" Jul 10 19:10:25.601192 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601157 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0"} err="failed to get container status \"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0\": rpc error: code = NotFound desc = could not find container \"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0\": container with ID starting with ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0 not found: ID does not exist" Jul 10 19:10:25.601192 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601169 2579 scope.go:117] "RemoveContainer" containerID="48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3" Jul 10 19:10:25.601391 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601370 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3"} err="failed to get container status \"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3\": rpc error: code = NotFound desc = could not find container \"48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3\": container with ID starting with 48dec3c8ee688cede0d0db0202e7c7be7a27ccf1f01b68fc975cbe5e4a3d3fb3 not found: ID does not exist" Jul 10 19:10:25.601461 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601394 2579 scope.go:117] "RemoveContainer" containerID="165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823" Jul 10 19:10:25.601620 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601602 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823"} err="failed to get container status \"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823\": rpc error: code = NotFound desc = could not find container \"165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823\": container with ID starting with 165b0a65978c036c8c09c99ec36cab709a69f344e54c5d2eb39887c4581dc823 not found: ID does not exist" Jul 10 19:10:25.601708 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601622 2579 scope.go:117] "RemoveContainer" containerID="65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea" Jul 10 19:10:25.601855 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601840 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea"} err="failed to get container status \"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea\": rpc error: code = NotFound desc = could not find container \"65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea\": container with ID starting with 65b5afdaee579a2fa9acd42631179d72c8c259f60d1a724cfca3a6b33868a7ea not found: ID does not exist" Jul 10 19:10:25.601912 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.601858 2579 scope.go:117] "RemoveContainer" containerID="faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949" Jul 10 19:10:25.602064 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.602043 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949"} err="failed to get container status \"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949\": rpc error: code = NotFound desc = could not find container \"faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949\": container with ID starting with faf39cee8c48e13e5c46ab5ad71697f87ba6ab5dae86624af806fd59ae48b949 not found: ID does not exist" Jul 10 19:10:25.602109 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.602066 2579 scope.go:117] "RemoveContainer" containerID="170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc" Jul 10 19:10:25.602260 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.602245 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc"} err="failed to get container status \"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc\": rpc error: code = NotFound desc = could not find container \"170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc\": container with ID starting with 170e39edaf9ea903ce9ce5a5d275f64e9fedbcc09aaf1ab3501a147d95d21afc not found: ID does not exist" Jul 10 19:10:25.602305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.602261 2579 scope.go:117] "RemoveContainer" containerID="6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5" Jul 10 19:10:25.602472 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.602451 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5"} err="failed to get container status \"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5\": rpc error: code = NotFound desc = could not find container \"6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5\": container with ID starting with 6a0f2535e62c8d12e25a7dd3c1bc7fd3b8f381cf463b68e0e649e53799f625e5 not found: ID does not exist" Jul 10 19:10:25.602472 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.602470 2579 scope.go:117] "RemoveContainer" containerID="ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0" Jul 10 19:10:25.602684 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.602640 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0"} err="failed to get container status \"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0\": rpc error: code = NotFound desc = could not find container \"ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0\": container with ID starting with ac83320350973679f8b87aebf5565ee613e004d406b1ad524b7d2975444232b0 not found: ID does not exist" Jul 10 19:10:25.636964 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.636945 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-trusted-ca-bundle\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637064 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.636973 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-config-volume\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637064 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.636995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637064 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637055 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637185 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637168 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-tls-assets\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637235 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637207 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-cluster-tls-config\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637286 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637338 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-config-out\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637387 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637343 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-web\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637387 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx4ph\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-kube-api-access-tx4ph\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637495 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637393 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:25.637495 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637420 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-metrics-client-ca\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637495 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637447 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-main-db\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.637495 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-web-config\") pod \"62f50804-74ce-4fd5-86fd-292270a0f51b\" (UID: \"62f50804-74ce-4fd5-86fd-292270a0f51b\") " Jul 10 19:10:25.638032 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.637808 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.638403 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.638373 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:10:25.638530 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.638508 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:25.640110 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.640084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:25.640413 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.640373 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:10:25.640611 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.640572 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:25.640991 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.640959 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-config-out" (OuterVolumeSpecName: "config-out") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:10:25.641075 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.641041 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-config-volume" (OuterVolumeSpecName: "config-volume") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:25.641138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.641112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:25.641347 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.641321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:25.641611 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.641582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-kube-api-access-tx4ph" (OuterVolumeSpecName: "kube-api-access-tx4ph") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "kube-api-access-tx4ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:10:25.644367 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.644339 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:25.650546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.650522 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-web-config" (OuterVolumeSpecName: "web-config") pod "62f50804-74ce-4fd5-86fd-292270a0f51b" (UID: "62f50804-74ce-4fd5-86fd-292270a0f51b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:25.738961 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.738931 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.738961 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.738960 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx4ph\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-kube-api-access-tx4ph\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.738961 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.738970 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62f50804-74ce-4fd5-86fd-292270a0f51b-metrics-client-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.738978 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-alertmanager-main-db\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.738988 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-web-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.738996 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-config-volume\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.739004 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.739012 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-main-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.739021 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62f50804-74ce-4fd5-86fd-292270a0f51b-tls-assets\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.739028 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-cluster-tls-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.739037 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62f50804-74ce-4fd5-86fd-292270a0f51b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.739147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.739047 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62f50804-74ce-4fd5-86fd-292270a0f51b-config-out\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:25.871486 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.871440 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:10:25.874786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.874768 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:10:25.899290 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899269 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:10:25.899553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899539 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="alertmanager" Jul 10 19:10:25.899596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899554 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="alertmanager" Jul 10 19:10:25.899596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899568 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="init-config-reloader" Jul 10 19:10:25.899596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899578 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="init-config-reloader" Jul 10 19:10:25.899596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899590 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c1563f-724f-4b56-becd-27544a07e426" containerName="console" Jul 10 19:10:25.899596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899595 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c1563f-724f-4b56-becd-27544a07e426" containerName="console" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899603 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899610 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899619 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="config-reloader" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899624 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="config-reloader" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899630 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="prom-label-proxy" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899642 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="prom-label-proxy" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899664 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-metric" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899673 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-metric" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899679 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="830feb7e-bb0d-42ba-a1fc-121b25d73ddf" containerName="console" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899685 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="830feb7e-bb0d-42ba-a1fc-121b25d73ddf" containerName="console" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899694 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-web" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899699 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-web" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899749 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="alertmanager" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899757 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899764 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-web" Jul 10 19:10:25.899764 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899771 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="prom-label-proxy" Jul 10 19:10:25.900232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899778 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c1563f-724f-4b56-becd-27544a07e426" containerName="console" Jul 10 19:10:25.900232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899783 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="config-reloader" Jul 10 19:10:25.900232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899788 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" containerName="kube-rbac-proxy-metric" Jul 10 19:10:25.900232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.899794 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="830feb7e-bb0d-42ba-a1fc-121b25d73ddf" containerName="console" Jul 10 19:10:25.904064 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.903502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:25.906046 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.906025 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 19:10:25.906331 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.906311 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 19:10:25.906613 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.906591 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 19:10:25.906721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.906620 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 19:10:25.906721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.906621 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 19:10:25.906721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.906620 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 19:10:25.907144 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.907079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 19:10:25.907254 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.907146 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 19:10:25.907450 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.907435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ptmpr\"" Jul 10 19:10:25.911491 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.911473 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 19:10:25.916020 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:25.916002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:10:26.041692 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041817 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-config-volume\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041817 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041817 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16e4ba2d-9c2b-4d65-a06f-9b067539826a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041817 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16e4ba2d-9c2b-4d65-a06f-9b067539826a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9t8\" (UniqueName: \"kubernetes.io/projected/16e4ba2d-9c2b-4d65-a06f-9b067539826a-kube-api-access-7l9t8\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16e4ba2d-9c2b-4d65-a06f-9b067539826a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/16e4ba2d-9c2b-4d65-a06f-9b067539826a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.041971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.042138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16e4ba2d-9c2b-4d65-a06f-9b067539826a-config-out\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.042138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.041995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.042138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.042059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-web-config\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.142843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.142787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9t8\" (UniqueName: \"kubernetes.io/projected/16e4ba2d-9c2b-4d65-a06f-9b067539826a-kube-api-access-7l9t8\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.142843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.142822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16e4ba2d-9c2b-4d65-a06f-9b067539826a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.142857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/16e4ba2d-9c2b-4d65-a06f-9b067539826a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.142970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16e4ba2d-9c2b-4d65-a06f-9b067539826a-config-out\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143369 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-web-config\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143369 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/16e4ba2d-9c2b-4d65-a06f-9b067539826a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143369 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143369 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-config-volume\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143369 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.143600 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.143387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16e4ba2d-9c2b-4d65-a06f-9b067539826a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.144174 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.144147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16e4ba2d-9c2b-4d65-a06f-9b067539826a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.144274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.144185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16e4ba2d-9c2b-4d65-a06f-9b067539826a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.144590 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.144567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16e4ba2d-9c2b-4d65-a06f-9b067539826a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.145945 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.145917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16e4ba2d-9c2b-4d65-a06f-9b067539826a-config-out\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.146080 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.146057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.146191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.146167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-web-config\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.146620 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.146588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.146736 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.146633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16e4ba2d-9c2b-4d65-a06f-9b067539826a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.146736 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.146673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.147032 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.147007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.147282 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.147264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-config-volume\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.147591 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.147575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16e4ba2d-9c2b-4d65-a06f-9b067539826a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.150442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.150425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9t8\" (UniqueName: \"kubernetes.io/projected/16e4ba2d-9c2b-4d65-a06f-9b067539826a-kube-api-access-7l9t8\") pod \"alertmanager-main-0\" (UID: \"16e4ba2d-9c2b-4d65-a06f-9b067539826a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.214014 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.213992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:10:26.347997 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.347972 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:10:26.350367 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:10:26.350329 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e4ba2d_9c2b_4d65_a06f_9b067539826a.slice/crio-3bf7cc74edb775ce86361614df1c8a4f18ea63321acb746f131bd350a7c395a4 WatchSource:0}: Error finding container 3bf7cc74edb775ce86361614df1c8a4f18ea63321acb746f131bd350a7c395a4: Status 404 returned error can't find the container with id 3bf7cc74edb775ce86361614df1c8a4f18ea63321acb746f131bd350a7c395a4 Jul 10 19:10:26.552574 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.552544 2579 generic.go:356] "Generic (PLEG): container finished" podID="16e4ba2d-9c2b-4d65-a06f-9b067539826a" containerID="5d25c7ca30de4dd80a2ccfafd957be99cc4e213a5b7d6c17476642b58a2a11ed" exitCode=0 Jul 10 19:10:26.552900 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.552586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerDied","Data":"5d25c7ca30de4dd80a2ccfafd957be99cc4e213a5b7d6c17476642b58a2a11ed"} Jul 10 19:10:26.552900 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:26.552663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerStarted","Data":"3bf7cc74edb775ce86361614df1c8a4f18ea63321acb746f131bd350a7c395a4"} Jul 10 19:10:27.070919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.070851 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f50804-74ce-4fd5-86fd-292270a0f51b" path="/var/lib/kubelet/pods/62f50804-74ce-4fd5-86fd-292270a0f51b/volumes" Jul 10 19:10:27.560030 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.559995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerStarted","Data":"54df53d3d523e50ddd61863828a6c6896dcb8dbb3ac9e60fca0196f106d1ff80"} Jul 10 19:10:27.560443 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.560057 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerStarted","Data":"dfe564902642c474d1b3b8ff8941f10216098babe75f210f36fa74d76584d6b7"} Jul 10 19:10:27.560443 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.560089 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerStarted","Data":"b6cda4ee7d20f429101aaeb7dce194c748ee77ebdc1d8e4eafc0791778be3b2e"} Jul 10 19:10:27.560443 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.560113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerStarted","Data":"2276d508069cd70d63ae70d8181a91d85035446ff980a54dd43171fe35100d0a"} Jul 10 19:10:27.560443 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.560145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerStarted","Data":"d3e8c3edb678709d8e5fe0da5b5940320a52c5ec9727ad44699964145f91c491"} Jul 10 19:10:27.560443 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.560169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16e4ba2d-9c2b-4d65-a06f-9b067539826a","Type":"ContainerStarted","Data":"68e0020dd532f700182f1bfe3aae63e046a0e43a2ca9a8fbf84f213291cef627"} Jul 10 19:10:27.587198 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:27.587154 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.572330033 podStartE2EDuration="2.587138728s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.808695ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:27.585804077 +0000 UTC m=+167.624677118" watchObservedRunningTime="2026-07-10 19:10:27.587138728 +0000 UTC m=+167.626011746" Jul 10 19:10:28.194723 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.194692 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f6775cdb4-8wpn6"] Jul 10 19:10:28.197773 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.197739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.200440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.200415 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 19:10:28.200570 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.200441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 19:10:28.200570 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.200500 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 19:10:28.200813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.200792 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 19:10:28.200921 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.200795 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 19:10:28.200921 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.200860 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vqmtf\"" Jul 10 19:10:28.206909 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.206876 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 19:10:28.209288 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.209269 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f6775cdb4-8wpn6"] Jul 10 19:10:28.260993 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.260963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-serving-certs-ca-bundle\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.261144 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.261005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-secret-telemeter-client\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.261144 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.261045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.261144 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.261070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxjp\" (UniqueName: \"kubernetes.io/projected/ddaba4d3-8e03-4800-b845-2bc0b535aa97-kube-api-access-prxjp\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.261144 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.261096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-federate-client-tls\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.261358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.261200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-metrics-client-ca\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.261358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.261240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.261358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.261278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-telemeter-client-tls\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.362666 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.362627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.362853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.362685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxjp\" (UniqueName: \"kubernetes.io/projected/ddaba4d3-8e03-4800-b845-2bc0b535aa97-kube-api-access-prxjp\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.362853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.362716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-federate-client-tls\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.362853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.362829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-metrics-client-ca\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.363012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.362874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.363012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.362910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-telemeter-client-tls\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.363012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.362961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-serving-certs-ca-bundle\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.363012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.363009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-secret-telemeter-client\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.363602 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.363573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-metrics-client-ca\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.364060 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.364033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.364161 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.364096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddaba4d3-8e03-4800-b845-2bc0b535aa97-serving-certs-ca-bundle\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.365431 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.365409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-federate-client-tls\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.365512 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.365426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-telemeter-client-tls\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.365609 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.365592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.365732 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.365715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddaba4d3-8e03-4800-b845-2bc0b535aa97-secret-telemeter-client\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.370959 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.370935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxjp\" (UniqueName: \"kubernetes.io/projected/ddaba4d3-8e03-4800-b845-2bc0b535aa97-kube-api-access-prxjp\") pod \"telemeter-client-f6775cdb4-8wpn6\" (UID: \"ddaba4d3-8e03-4800-b845-2bc0b535aa97\") " pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.499049 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.498954 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:10:28.499702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.499318 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-thanos" containerID="cri-o://13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3" gracePeriod=600 Jul 10 19:10:28.499702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.499318 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="prometheus" containerID="cri-o://4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" gracePeriod=600 Jul 10 19:10:28.499702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.499359 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-web" containerID="cri-o://fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b" gracePeriod=600 Jul 10 19:10:28.499702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.499405 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="thanos-sidecar" containerID="cri-o://b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0" gracePeriod=600 Jul 10 19:10:28.499702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.499463 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="config-reloader" containerID="cri-o://ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf" gracePeriod=600 Jul 10 19:10:28.499702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.499522 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy" containerID="cri-o://ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821" gracePeriod=600 Jul 10 19:10:28.509578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.508831 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" Jul 10 19:10:28.670581 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:28.670480 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f6775cdb4-8wpn6"] Jul 10 19:10:28.673033 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:10:28.673010 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaba4d3_8e03_4800_b845_2bc0b535aa97.slice/crio-31cc596b9cc8636d43059ddfdcacc73707e1decbe4533d654bf68f8fa30505f0 WatchSource:0}: Error finding container 31cc596b9cc8636d43059ddfdcacc73707e1decbe4533d654bf68f8fa30505f0: Status 404 returned error can't find the container with id 31cc596b9cc8636d43059ddfdcacc73707e1decbe4533d654bf68f8fa30505f0 Jul 10 19:10:29.576970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:29.576938 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerID="fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b" exitCode=0 Jul 10 19:10:29.576970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:29.576967 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerID="b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0" exitCode=0 Jul 10 19:10:29.577130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:29.576976 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerID="ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf" exitCode=0 Jul 10 19:10:29.577130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:29.577032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b"} Jul 10 19:10:29.577130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:29.577099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0"} Jul 10 19:10:29.577301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:29.577137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf"} Jul 10 19:10:29.578722 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:29.578638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" event={"ID":"ddaba4d3-8e03-4800-b845-2bc0b535aa97","Type":"ContainerStarted","Data":"31cc596b9cc8636d43059ddfdcacc73707e1decbe4533d654bf68f8fa30505f0"} Jul 10 19:10:30.588198 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:30.588167 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerID="13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3" exitCode=0 Jul 10 19:10:30.588198 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:30.588192 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerID="ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821" exitCode=0 Jul 10 19:10:30.588729 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:30.588229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3"} Jul 10 19:10:30.588729 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:30.588304 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821"} Jul 10 19:10:31.486755 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:31.486719 2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 10 19:10:31.487648 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:31.487615 2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 10 19:10:31.488403 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:31.488376 2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Jul 10 19:10:31.488468 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:31.488413 2579 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="prometheus" probeResult="unknown" Jul 10 19:10:31.596135 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:31.596106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" event={"ID":"ddaba4d3-8e03-4800-b845-2bc0b535aa97","Type":"ContainerStarted","Data":"7eb7614485c6c1646dabff45923c5588c6fcda6881a50181d61c95fdb0640ee0"} Jul 10 19:10:31.596529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:31.596163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" event={"ID":"ddaba4d3-8e03-4800-b845-2bc0b535aa97","Type":"ContainerStarted","Data":"19a8be0ac393925fe3c82f87524bf1a5e66a8d190170ec23a27df1e5a86ac9f2"} Jul 10 19:10:31.596529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:31.596190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" event={"ID":"ddaba4d3-8e03-4800-b845-2bc0b535aa97","Type":"ContainerStarted","Data":"ec4a95c0ce0c1ff92b0f6f5bb191385b2c1ad2ced7cba1ec99d6d6414c93c799"} Jul 10 19:10:31.619297 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:31.619246 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f6775cdb4-8wpn6" podStartSLOduration=1.5003317520000001 podStartE2EDuration="3.619228647s" totalImagesPullingTime="2.118896895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:31.616389974 +0000 UTC m=+171.655263008" watchObservedRunningTime="2026-07-10 19:10:31.619228647 +0000 UTC m=+171.658101665" Jul 10 19:10:32.473889 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:32.473792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:32.528604 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:32.528553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:36.455493 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.455469 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.533137 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533075 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533137 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533104 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533137 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533131 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nslx\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-kube-api-access-6nslx\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-trusted-ca-bundle\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533171 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-thanos-prometheus-http-client-file\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533199 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-2\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533229 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-tls\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-config\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533301 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-config-out\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533326 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-metrics-client-ca\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533367 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-0\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533400 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-metrics-client-certs\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-kubelet-serving-ca-bundle\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.533469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533472 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-tls-assets\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533505 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-web-config\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533529 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533538 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-grpc-tls\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-serving-certs-ca-bundle\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533589 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-1\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533671 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-kube-rbac-proxy\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.534045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.533698 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-db\") pod \"1bad7f1c-26e6-4232-a047-ad133bd255cf\" (UID: \"1bad7f1c-26e6-4232-a047-ad133bd255cf\") " Jul 10 19:10:36.534442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.534139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:36.534442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.534249 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.534442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.534271 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.534442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.534289 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-metrics-client-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.534442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.534341 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:36.534803 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.534776 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:36.535473 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.535438 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:10:36.536077 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.536046 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:36.537274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537247 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.537376 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-config" (OuterVolumeSpecName: "config") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.537441 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-kube-api-access-6nslx" (OuterVolumeSpecName: "kube-api-access-6nslx") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "kube-api-access-6nslx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:10:36.537441 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537418 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.537523 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.537563 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537517 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.537563 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537545 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:10:36.537967 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.537943 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.538048 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.538021 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-config-out" (OuterVolumeSpecName: "config-out") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:10:36.538383 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.538357 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.538926 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.538896 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.539213 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.539194 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:10:36.548644 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.548621 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-web-config" (OuterVolumeSpecName: "web-config") pod "1bad7f1c-26e6-4232-a047-ad133bd255cf" (UID: "1bad7f1c-26e6-4232-a047-ad133bd255cf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:10:36.622456 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.622427 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerID="4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" exitCode=0 Jul 10 19:10:36.622560 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.622474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c"} Jul 10 19:10:36.622560 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.622516 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.622560 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.622545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1bad7f1c-26e6-4232-a047-ad133bd255cf","Type":"ContainerDied","Data":"24a9e88cad51b65f2a18b64f8891ab23e6e760f9cf8ad67e76983773368ead48"} Jul 10 19:10:36.622668 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.622593 2579 scope.go:117] "RemoveContainer" containerID="13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3" Jul 10 19:10:36.629942 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.629922 2579 scope.go:117] "RemoveContainer" containerID="ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821" Jul 10 19:10:36.635139 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635118 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-tls-assets\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635141 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-web-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635153 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-grpc-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635167 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635180 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635193 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-kube-rbac-proxy\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635207 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-db\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635220 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635234 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635248 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nslx\" (UniqueName: \"kubernetes.io/projected/1bad7f1c-26e6-4232-a047-ad133bd255cf-kube-api-access-6nslx\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635262 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635278 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635290 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-config\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635303 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad7f1c-26e6-4232-a047-ad133bd255cf-config-out\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635318 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635335 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1bad7f1c-26e6-4232-a047-ad133bd255cf-secret-metrics-client-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.635524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.635349 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad7f1c-26e6-4232-a047-ad133bd255cf-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:10:36.636584 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.636565 2579 scope.go:117] "RemoveContainer" containerID="fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b" Jul 10 19:10:36.642928 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.642909 2579 scope.go:117] "RemoveContainer" containerID="b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0" Jul 10 19:10:36.647606 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.647588 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:10:36.650424 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.650408 2579 scope.go:117] "RemoveContainer" containerID="ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf" Jul 10 19:10:36.651982 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.651961 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:10:36.656713 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.656698 2579 scope.go:117] "RemoveContainer" containerID="4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" Jul 10 19:10:36.663308 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.663292 2579 scope.go:117] "RemoveContainer" containerID="57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb" Jul 10 19:10:36.669868 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.669852 2579 scope.go:117] "RemoveContainer" containerID="13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3" Jul 10 19:10:36.670111 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:36.670095 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3\": container with ID starting with 13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3 not found: ID does not exist" containerID="13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3" Jul 10 19:10:36.670160 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670118 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3"} err="failed to get container status \"13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3\": rpc error: code = NotFound desc = could not find container \"13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3\": container with ID starting with 13e9e3b9c80c343cf323fd4bcdf84fb86b0d3c46f775ac7cc20a71bd76256cb3 not found: ID does not exist" Jul 10 19:10:36.670160 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670134 2579 scope.go:117] "RemoveContainer" containerID="ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821" Jul 10 19:10:36.670368 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:36.670351 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821\": container with ID starting with ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821 not found: ID does not exist" containerID="ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821" Jul 10 19:10:36.670430 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670376 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821"} err="failed to get container status \"ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821\": rpc error: code = NotFound desc = could not find container \"ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821\": container with ID starting with ceadd86fd0f05391df357f278ff57ebc7d07064a3f9d98c21229a46857ff3821 not found: ID does not exist" Jul 10 19:10:36.670430 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670398 2579 scope.go:117] "RemoveContainer" containerID="fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b" Jul 10 19:10:36.670607 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:36.670590 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b\": container with ID starting with fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b not found: ID does not exist" containerID="fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b" Jul 10 19:10:36.670645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670611 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b"} err="failed to get container status \"fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b\": rpc error: code = NotFound desc = could not find container \"fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b\": container with ID starting with fa0fec840caf1a442713e58ba589c16447aa64526a5d2c9de8bc9191ac03045b not found: ID does not exist" Jul 10 19:10:36.670645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670627 2579 scope.go:117] "RemoveContainer" containerID="b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0" Jul 10 19:10:36.670852 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:36.670833 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0\": container with ID starting with b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0 not found: ID does not exist" containerID="b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0" Jul 10 19:10:36.670905 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670857 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0"} err="failed to get container status \"b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0\": rpc error: code = NotFound desc = could not find container \"b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0\": container with ID starting with b7fe7e8f4abf4dfb4044c66b7c507f288477f0e5434b099df5fb70faac7007c0 not found: ID does not exist" Jul 10 19:10:36.670905 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.670873 2579 scope.go:117] "RemoveContainer" containerID="ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf" Jul 10 19:10:36.671089 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:36.671070 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf\": container with ID starting with ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf not found: ID does not exist" containerID="ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf" Jul 10 19:10:36.671131 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.671094 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf"} err="failed to get container status \"ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf\": rpc error: code = NotFound desc = could not find container \"ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf\": container with ID starting with ddce7559307e4ead2f81bd9128f28a37e5135ebad432ee0c92d7994243840fcf not found: ID does not exist" Jul 10 19:10:36.671131 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.671108 2579 scope.go:117] "RemoveContainer" containerID="4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" Jul 10 19:10:36.671328 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:36.671313 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c\": container with ID starting with 4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c not found: ID does not exist" containerID="4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c" Jul 10 19:10:36.671371 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.671331 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c"} err="failed to get container status \"4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c\": rpc error: code = NotFound desc = could not find container \"4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c\": container with ID starting with 4267e5f9282409c366f783efb008b714bc0747b5fa85d2c8efbe720739a2007c not found: ID does not exist" Jul 10 19:10:36.671371 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.671345 2579 scope.go:117] "RemoveContainer" containerID="57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb" Jul 10 19:10:36.671552 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:10:36.671535 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb\": container with ID starting with 57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb not found: ID does not exist" containerID="57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb" Jul 10 19:10:36.671599 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.671560 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb"} err="failed to get container status \"57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb\": rpc error: code = NotFound desc = could not find container \"57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb\": container with ID starting with 57722594a1b8b6e6c3f50b15382a12f4f48cd65a574ff260b7fb4a30edd091bb not found: ID does not exist" Jul 10 19:10:36.676099 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676082 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:10:36.676358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676347 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="prometheus" Jul 10 19:10:36.676399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676360 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="prometheus" Jul 10 19:10:36.676399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676369 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-web" Jul 10 19:10:36.676399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676375 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-web" Jul 10 19:10:36.676399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676386 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="init-config-reloader" Jul 10 19:10:36.676399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676391 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="init-config-reloader" Jul 10 19:10:36.676399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676399 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="config-reloader" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676404 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="config-reloader" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676410 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676415 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676422 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="thanos-sidecar" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676427 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="thanos-sidecar" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676439 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-thanos" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676444 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-thanos" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676492 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-web" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676500 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="thanos-sidecar" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676507 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="config-reloader" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676513 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="prometheus" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676518 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy" Jul 10 19:10:36.676612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.676523 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" containerName="kube-rbac-proxy-thanos" Jul 10 19:10:36.680804 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.680790 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.683507 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.683488 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 19:10:36.683507 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.683499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 19:10:36.683802 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.683786 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 19:10:36.683897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.683811 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 19:10:36.683897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.683867 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 19:10:36.684000 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.683935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 19:10:36.684054 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.684023 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 19:10:36.684135 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.684097 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 19:10:36.684476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.684460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 19:10:36.684557 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.684501 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 19:10:36.685165 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.685144 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 19:10:36.685287 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.685167 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qlfr5\"" Jul 10 19:10:36.685287 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.685253 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-atsv3quggohf\"" Jul 10 19:10:36.685402 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.685297 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 19:10:36.687350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.687329 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 19:10:36.690981 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.690964 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 19:10:36.695132 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.695114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:10:36.836461 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836569 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836569 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836569 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836694 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836694 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/677af96a-5af1-468b-a579-a91cd84b46ca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836694 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836694 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836815 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836815 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836815 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836904 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836904 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836904 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dj8g\" (UniqueName: \"kubernetes.io/projected/677af96a-5af1-468b-a579-a91cd84b46ca-kube-api-access-7dj8g\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.836904 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-config\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.837025 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/677af96a-5af1-468b-a579-a91cd84b46ca-config-out\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.837025 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.837025 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-web-config\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.837025 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.836972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.837025 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.837003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.937996 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.937973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938094 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938094 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938094 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dj8g\" (UniqueName: \"kubernetes.io/projected/677af96a-5af1-468b-a579-a91cd84b46ca-kube-api-access-7dj8g\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938201 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-config\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938234 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/677af96a-5af1-468b-a579-a91cd84b46ca-config-out\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938280 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938322 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-web-config\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938376 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938376 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/677af96a-5af1-468b-a579-a91cd84b46ca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.938717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.939019 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.938740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.939427 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.939401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.939523 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.939510 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.941000 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.940967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/677af96a-5af1-468b-a579-a91cd84b46ca-config-out\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.941111 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.941072 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.941813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.941191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-config\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.941813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.941232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.941813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.941271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.941813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.941708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.942084 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.941821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.942165 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.942135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.942395 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.942378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/677af96a-5af1-468b-a579-a91cd84b46ca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.942464 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.942427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.942771 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.942747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.942881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.942859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.943285 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.943264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.943477 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.943461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-web-config\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.944598 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.944577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/677af96a-5af1-468b-a579-a91cd84b46ca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.944898 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.944881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/677af96a-5af1-468b-a579-a91cd84b46ca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.946732 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.946717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dj8g\" (UniqueName: \"kubernetes.io/projected/677af96a-5af1-468b-a579-a91cd84b46ca-kube-api-access-7dj8g\") pod \"prometheus-k8s-0\" (UID: \"677af96a-5af1-468b-a579-a91cd84b46ca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:36.990931 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:36.990899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:37.071145 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:37.071117 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bad7f1c-26e6-4232-a047-ad133bd255cf" path="/var/lib/kubelet/pods/1bad7f1c-26e6-4232-a047-ad133bd255cf/volumes" Jul 10 19:10:37.139904 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:37.139272 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:10:37.142855 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:10:37.142830 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677af96a_5af1_468b_a579_a91cd84b46ca.slice/crio-bd31b3964ffde225357fd4c07d7e1b5047252f75711bfca504e99950fef33e26 WatchSource:0}: Error finding container bd31b3964ffde225357fd4c07d7e1b5047252f75711bfca504e99950fef33e26: Status 404 returned error can't find the container with id bd31b3964ffde225357fd4c07d7e1b5047252f75711bfca504e99950fef33e26 Jul 10 19:10:37.627867 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:37.627838 2579 generic.go:356] "Generic (PLEG): container finished" podID="677af96a-5af1-468b-a579-a91cd84b46ca" containerID="4766a6dddf660058c1075040eb943adb940bdb1af31f3a3109a4514350d321d2" exitCode=0 Jul 10 19:10:37.628296 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:37.627885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerDied","Data":"4766a6dddf660058c1075040eb943adb940bdb1af31f3a3109a4514350d321d2"} Jul 10 19:10:37.628296 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:37.627934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerStarted","Data":"bd31b3964ffde225357fd4c07d7e1b5047252f75711bfca504e99950fef33e26"} Jul 10 19:10:38.635348 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:38.635317 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerStarted","Data":"e1f22a1d1be4bdcdd455334309824de83021832ece99267509b4f28f422141ca"} Jul 10 19:10:38.635737 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:38.635383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerStarted","Data":"5ddfb84a911baac4c73f094b631c4c1afb89a78b0cff0679a96c13aad3f7a22c"} Jul 10 19:10:38.635737 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:38.635408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerStarted","Data":"0e9227effa98a7f80bb1b2084d27ad775ebe56fcc7bd374779b2ceb2cb5c23a4"} Jul 10 19:10:38.635737 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:38.635446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerStarted","Data":"16ced7b513b65d3176e727ba23d9da3d1f883c7426809743c000df5bb32baaf6"} Jul 10 19:10:38.635737 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:38.635481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerStarted","Data":"f14a663e850b958e7ae077087815c1a9ccaf80966d5a66f55db1116555bd4c50"} Jul 10 19:10:38.635737 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:38.635508 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"677af96a-5af1-468b-a579-a91cd84b46ca","Type":"ContainerStarted","Data":"b89a0156a92b12c43e63f0ddb8be8fc69823457bc29aa7aed7c0f4291cb22f46"} Jul 10 19:10:38.665860 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:38.665809 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.648544696 podStartE2EDuration="2.665794559s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.249863ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:38.663936472 +0000 UTC m=+178.702809508" watchObservedRunningTime="2026-07-10 19:10:38.665794559 +0000 UTC m=+178.704667701" Jul 10 19:10:40.296995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:40.296957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:41.991023 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:41.990992 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:10:43.586058 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:43.586011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:43.687333 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:43.687291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:43.739832 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:43.739789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:50.740866 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:50.740832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:55.895101 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:55.895050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:55.959853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:10:55.959826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:36.991030 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:36.991000 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:11:37.006614 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:37.006585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:11:37.877283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:37.877258 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:11:38.392343 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:38.392302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:38.500594 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:38.500565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:48.177837 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:48.177803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:48.257160 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:48.257112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:50.407147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:50.407118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:50.477565 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:50.477537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:58.892918 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:58.892878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:58.993527 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:58.993495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:59.094361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:11:59.094328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:06.668910 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:12:06.668821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:06.768793 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:12:06.768749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:37.554006 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:12:37.553974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:37.638871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:12:37.638835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:40.844405 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:12:40.844370 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:12:40.844940 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:12:40.844426 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:12:59.384263 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:12:59.384227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:00.536088 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:00.536048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:03.187467 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:03.187435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:03.288980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:03.288947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:05.111221 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:05.111188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:05.212433 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:05.212388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:05.409108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:05.409019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:05.520863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:05.520832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:07.500562 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:07.500528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:07.710411 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:07.710384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:10.182221 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:10.182184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:10.282489 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:10.282456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:14.720335 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:14.720294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:21.604052 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:21.604010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:21.697812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:21.697782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:28.969771 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:28.969738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:29.072590 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:29.072559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:30.863647 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:30.863607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:34.061692 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:34.061593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:34.127371 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:34.127340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:37.724471 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:37.724439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:42.747856 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:42.747820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:42.909647 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:42.909614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:43.300603 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.300574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:43.494098 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.494060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:43.568947 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.568881 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747"] Jul 10 19:13:43.571331 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.571315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.579853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.579836 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 19:13:43.581069 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.581053 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5v77z\"" Jul 10 19:13:43.581163 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.581085 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 19:13:43.581163 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.581090 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 19:13:43.583005 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.582991 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 19:13:43.600553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.600529 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:13:43.605847 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.605827 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747"] Jul 10 19:13:43.637631 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.637607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/681431d1-a903-49f2-bf27-80c9a909982d-manager-config\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.637733 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.637635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/681431d1-a903-49f2-bf27-80c9a909982d-metrics-cert\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.637733 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.637688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/681431d1-a903-49f2-bf27-80c9a909982d-cert\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.637733 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.637714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm59c\" (UniqueName: \"kubernetes.io/projected/681431d1-a903-49f2-bf27-80c9a909982d-kube-api-access-wm59c\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.738863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.738838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/681431d1-a903-49f2-bf27-80c9a909982d-manager-config\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.738956 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.738867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/681431d1-a903-49f2-bf27-80c9a909982d-metrics-cert\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.738956 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.738901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/681431d1-a903-49f2-bf27-80c9a909982d-cert\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.739033 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.739007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm59c\" (UniqueName: \"kubernetes.io/projected/681431d1-a903-49f2-bf27-80c9a909982d-kube-api-access-wm59c\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.739624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.739608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/681431d1-a903-49f2-bf27-80c9a909982d-manager-config\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.741329 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.741312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/681431d1-a903-49f2-bf27-80c9a909982d-metrics-cert\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.741390 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.741371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/681431d1-a903-49f2-bf27-80c9a909982d-cert\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.773324 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.773296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm59c\" (UniqueName: \"kubernetes.io/projected/681431d1-a903-49f2-bf27-80c9a909982d-kube-api-access-wm59c\") pod \"lws-controller-manager-5d6cfdfcb9-dd747\" (UID: \"681431d1-a903-49f2-bf27-80c9a909982d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:43.861616 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.861591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:43.879567 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:43.879542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:44.029986 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.029962 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747"] Jul 10 19:13:44.032111 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:13:44.032083 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681431d1_a903_49f2_bf27_80c9a909982d.slice/crio-9cb2759025293d2f76a13569832c51cd6fb543ec03734b16cb63326d26e3f560 WatchSource:0}: Error finding container 9cb2759025293d2f76a13569832c51cd6fb543ec03734b16cb63326d26e3f560: Status 404 returned error can't find the container with id 9cb2759025293d2f76a13569832c51cd6fb543ec03734b16cb63326d26e3f560 Jul 10 19:13:44.034028 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.034012 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:13:44.140377 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.140301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:44.267260 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.267224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:44.343987 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.343957 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" event={"ID":"681431d1-a903-49f2-bf27-80c9a909982d","Type":"ContainerStarted","Data":"9cb2759025293d2f76a13569832c51cd6fb543ec03734b16cb63326d26e3f560"} Jul 10 19:13:44.352228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.352198 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s"] Jul 10 19:13:44.355135 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.355119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.358841 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.358822 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 19:13:44.358936 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.358899 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 19:13:44.359037 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.359022 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 19:13:44.359285 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.359272 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4cjwg\"" Jul 10 19:13:44.365374 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.365357 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 19:13:44.385819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.385797 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s"] Jul 10 19:13:44.444265 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.444209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxfj\" (UniqueName: \"kubernetes.io/projected/f399cf7a-6480-4d4a-8771-985942ec039d-kube-api-access-bbxfj\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.444265 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.444248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f399cf7a-6480-4d4a-8771-985942ec039d-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.444382 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.444332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f399cf7a-6480-4d4a-8771-985942ec039d-manifests\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.444382 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.444356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f399cf7a-6480-4d4a-8771-985942ec039d-tmp\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.444452 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.444384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f399cf7a-6480-4d4a-8771-985942ec039d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.545301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.545280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f399cf7a-6480-4d4a-8771-985942ec039d-manifests\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.545395 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.545307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f399cf7a-6480-4d4a-8771-985942ec039d-tmp\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.545395 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.545327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f399cf7a-6480-4d4a-8771-985942ec039d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.545395 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.545357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxfj\" (UniqueName: \"kubernetes.io/projected/f399cf7a-6480-4d4a-8771-985942ec039d-kube-api-access-bbxfj\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.545521 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.545403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f399cf7a-6480-4d4a-8771-985942ec039d-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.545721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.545701 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f399cf7a-6480-4d4a-8771-985942ec039d-manifests\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.547510 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.547485 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f399cf7a-6480-4d4a-8771-985942ec039d-tmp\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.547750 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.547731 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f399cf7a-6480-4d4a-8771-985942ec039d-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.547867 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.547848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f399cf7a-6480-4d4a-8771-985942ec039d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.558150 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.558130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxfj\" (UniqueName: \"kubernetes.io/projected/f399cf7a-6480-4d4a-8771-985942ec039d-kube-api-access-bbxfj\") pod \"opendatahub-operator-controller-manager-64d686dc7f-7j97s\" (UID: \"f399cf7a-6480-4d4a-8771-985942ec039d\") " pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.618199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.618176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:44.665892 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.665871 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:44.719521 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.719497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:44.803422 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.803289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s"] Jul 10 19:13:44.805534 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:13:44.805497 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf399cf7a_6480_4d4a_8771_985942ec039d.slice/crio-4b933dacf85e75b56f2a5809fa20bd27e44c3ef749d3b2d3e2a48f595c4202cf WatchSource:0}: Error finding container 4b933dacf85e75b56f2a5809fa20bd27e44c3ef749d3b2d3e2a48f595c4202cf: Status 404 returned error can't find the container with id 4b933dacf85e75b56f2a5809fa20bd27e44c3ef749d3b2d3e2a48f595c4202cf Jul 10 19:13:44.806223 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.806185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:44.902319 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:44.902290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.003455 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:45.003396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.152514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:45.152490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.349712 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:45.349686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" event={"ID":"f399cf7a-6480-4d4a-8771-985942ec039d","Type":"ContainerStarted","Data":"4b933dacf85e75b56f2a5809fa20bd27e44c3ef749d3b2d3e2a48f595c4202cf"} Jul 10 19:13:48.374202 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:48.374163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" event={"ID":"f399cf7a-6480-4d4a-8771-985942ec039d","Type":"ContainerStarted","Data":"4787bea79553afef8be58d07d970fd07440282142b4ac6a1453f8e862317111a"} Jul 10 19:13:49.379730 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:49.379698 2579 generic.go:356] "Generic (PLEG): container finished" podID="f399cf7a-6480-4d4a-8771-985942ec039d" containerID="4787bea79553afef8be58d07d970fd07440282142b4ac6a1453f8e862317111a" exitCode=0 Jul 10 19:13:49.380081 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:49.379747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" event={"ID":"f399cf7a-6480-4d4a-8771-985942ec039d","Type":"ContainerDied","Data":"4787bea79553afef8be58d07d970fd07440282142b4ac6a1453f8e862317111a"} Jul 10 19:13:50.386551 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:50.386518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" event={"ID":"f399cf7a-6480-4d4a-8771-985942ec039d","Type":"ContainerStarted","Data":"e5964a6dab53f7bc0f581f617eb0ade8946aa2d6f5bf5506cee53b489ea78892"} Jul 10 19:13:50.386979 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:50.386632 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:13:50.411612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:50.411565 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" podStartSLOduration=2.013033765 podStartE2EDuration="6.411543102s" totalImagesPullingTime="3.371339121s" totalInitContainerRuntime="1.027170216s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:13:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:13:50.410734181 +0000 UTC m=+370.449607217" watchObservedRunningTime="2026-07-10 19:13:50.411543102 +0000 UTC m=+370.450416118" Jul 10 19:13:54.407533 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:54.407498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" event={"ID":"681431d1-a903-49f2-bf27-80c9a909982d","Type":"ContainerStarted","Data":"dda7e542e12e2eac1a73f13a715fda9e3d1f452ff3602e247c238f69e67c6146"} Jul 10 19:13:54.407929 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:54.407573 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:13:54.425957 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:54.425914 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" podStartSLOduration=1.8663135560000002 podStartE2EDuration="11.425900517s" totalImagesPullingTime="9.559586961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:13:54.424640133 +0000 UTC m=+374.463513151" watchObservedRunningTime="2026-07-10 19:13:54.425900517 +0000 UTC m=+374.464773534" Jul 10 19:13:56.179153 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:56.179121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:56.244109 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:56.244081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:56.615919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:56.615890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:56.715498 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:13:56.715477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:01.392399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:01.392373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64d686dc7f-7j97s" Jul 10 19:14:03.865280 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:03.865245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:04.054268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.054234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:04.485197 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.485150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:04.586202 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.586172 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-967644b46-ckv89"] Jul 10 19:14:04.588564 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.588547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.591840 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.591822 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 19:14:04.592061 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.592046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-plmhf\"" Jul 10 19:14:04.593050 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.593033 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 19:14:04.593109 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.593073 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 19:14:04.593109 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.593083 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 19:14:04.611222 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.611200 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-967644b46-ckv89"] Jul 10 19:14:04.713140 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.713114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpvx\" (UniqueName: \"kubernetes.io/projected/6eb62fa8-498a-4445-bf23-b5389bd56434-kube-api-access-knpvx\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.713250 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.713178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eb62fa8-498a-4445-bf23-b5389bd56434-tmp\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.713250 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.713221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb62fa8-498a-4445-bf23-b5389bd56434-tls-certs\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.813846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.813791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eb62fa8-498a-4445-bf23-b5389bd56434-tmp\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.813846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.813823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb62fa8-498a-4445-bf23-b5389bd56434-tls-certs\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.813976 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.813874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpvx\" (UniqueName: \"kubernetes.io/projected/6eb62fa8-498a-4445-bf23-b5389bd56434-kube-api-access-knpvx\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.814012 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:14:04.813992 2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 10 19:14:04.814069 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:14:04.814059 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eb62fa8-498a-4445-bf23-b5389bd56434-tls-certs podName:6eb62fa8-498a-4445-bf23-b5389bd56434 nodeName:}" failed. No retries permitted until 2026-07-10 19:14:05.314039674 +0000 UTC m=+385.352912685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6eb62fa8-498a-4445-bf23-b5389bd56434-tls-certs") pod "kube-auth-proxy-967644b46-ckv89" (UID: "6eb62fa8-498a-4445-bf23-b5389bd56434") : secret "kube-auth-proxy-tls" not found Jul 10 19:14:04.815924 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.815900 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eb62fa8-498a-4445-bf23-b5389bd56434-tmp\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:04.823739 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:04.823719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpvx\" (UniqueName: \"kubernetes.io/projected/6eb62fa8-498a-4445-bf23-b5389bd56434-kube-api-access-knpvx\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:05.010812 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.010779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:05.114565 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.114532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:05.242380 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.242330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:05.318942 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.318910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb62fa8-498a-4445-bf23-b5389bd56434-tls-certs\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:05.321279 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.321251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb62fa8-498a-4445-bf23-b5389bd56434-tls-certs\") pod \"kube-auth-proxy-967644b46-ckv89\" (UID: \"6eb62fa8-498a-4445-bf23-b5389bd56434\") " pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:05.360626 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.360601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:05.413704 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.413635 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d6cfdfcb9-dd747" Jul 10 19:14:05.497345 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.497312 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" Jul 10 19:14:05.640454 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:14:05.640422 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb62fa8_498a_4445_bf23_b5389bd56434.slice/crio-227c22259faefe6a79f5cdb407359f01206df2129a0c77341c9d219d8857550a WatchSource:0}: Error finding container 227c22259faefe6a79f5cdb407359f01206df2129a0c77341c9d219d8857550a: Status 404 returned error can't find the container with id 227c22259faefe6a79f5cdb407359f01206df2129a0c77341c9d219d8857550a Jul 10 19:14:05.640648 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.640624 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-967644b46-ckv89"] Jul 10 19:14:05.987363 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:05.987331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:06.220083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:06.220034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:06.322247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:06.322169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:06.419789 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:06.419760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:06.458166 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:06.458138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" event={"ID":"6eb62fa8-498a-4445-bf23-b5389bd56434","Type":"ContainerStarted","Data":"227c22259faefe6a79f5cdb407359f01206df2129a0c77341c9d219d8857550a"} Jul 10 19:14:07.581343 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:07.581313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:09.472971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:09.472937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" event={"ID":"6eb62fa8-498a-4445-bf23-b5389bd56434","Type":"ContainerStarted","Data":"6306595a4154efffc54dbfdeb6be240de6efcf51b4bc5e9ef10d3107ce9226a8"} Jul 10 19:14:09.493275 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:09.493231 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-967644b46-ckv89" podStartSLOduration=2.193812265 podStartE2EDuration="5.493215162s" totalImagesPullingTime="3.299402897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:14:09.490945507 +0000 UTC m=+389.529818563" watchObservedRunningTime="2026-07-10 19:14:09.493215162 +0000 UTC m=+389.532088204" Jul 10 19:14:13.493036 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:13.492990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:13.589096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:13.589063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:13.896134 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:13.896099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:14.051768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:14.051728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:17.168026 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:17.167997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:17.289063 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:17.289015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:17.563778 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:17.563702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:17.668849 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:17.668822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:18.494962 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:18.494933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:18.696022 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:18.695991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:18.843500 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:18.843474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:19.294430 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:19.294358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:19.452814 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:19.452786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:58.470175 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:58.470143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:58.544867 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:58.544833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:59.141486 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:59.141455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:59.210645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:14:59.210614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:05.581215 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:05.581129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:05.678789 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:05.678765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:19.589754 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:19.589719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:20.420491 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:20.420457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:21.181909 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:21.181879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:23.478236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:23.478201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:23.602762 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:23.602729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:24.423443 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:24.423413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:24.819353 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:24.819281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:25.109720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:25.109690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:27.577995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:27.577965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:29.103312 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:29.103272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:29.796167 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:29.796134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:30.460805 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:30.460775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:31.126009 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:31.125979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:31.242705 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:31.242679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:32.166121 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.166091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:32.173923 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.173896 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9"] Jul 10 19:15:32.177056 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.177041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.180217 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.180198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 19:15:32.180499 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.180483 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 19:15:32.180576 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.180557 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 19:15:32.181689 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.181667 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 19:15:32.181802 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.181695 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cmfwj\"" Jul 10 19:15:32.187376 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.187357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9"] Jul 10 19:15:32.281887 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.281861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.281996 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.281914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.281996 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.281957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9lk\" (UniqueName: \"kubernetes.io/projected/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-kube-api-access-bw9lk\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.339774 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.339744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:32.383047 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.383021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.383164 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.383057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9lk\" (UniqueName: \"kubernetes.io/projected/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-kube-api-access-bw9lk\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.383164 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.383124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.383755 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.383733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.385366 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.385346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.391875 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.391853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9lk\" (UniqueName: \"kubernetes.io/projected/293eb0dd-0cdf-432d-95f4-8997f3aeaeec-kube-api-access-bw9lk\") pod \"kuadrant-console-plugin-85fd4b8b9d-9gxd9\" (UID: \"293eb0dd-0cdf-432d-95f4-8997f3aeaeec\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.486441 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.486394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" Jul 10 19:15:32.636444 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.636409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9"] Jul 10 19:15:32.640441 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:15:32.640413 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod293eb0dd_0cdf_432d_95f4_8997f3aeaeec.slice/crio-f2f4675ff158972f37f0f3bc9ac6939ab80d09b3bb6d02a85851ef429399714e WatchSource:0}: Error finding container f2f4675ff158972f37f0f3bc9ac6939ab80d09b3bb6d02a85851ef429399714e: Status 404 returned error can't find the container with id f2f4675ff158972f37f0f3bc9ac6939ab80d09b3bb6d02a85851ef429399714e Jul 10 19:15:32.771889 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.771820 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd6dbb547-np9wr"] Jul 10 19:15:32.777673 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.777618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.782333 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.782303 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 19:15:32.782452 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.782358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 19:15:32.782452 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.782428 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9xtw5\"" Jul 10 19:15:32.782527 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.782358 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 19:15:32.783706 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.783687 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 19:15:32.784074 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.784060 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 19:15:32.789097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.789080 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 19:15:32.794788 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.794769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd6dbb547-np9wr"] Jul 10 19:15:32.829762 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.829736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" event={"ID":"293eb0dd-0cdf-432d-95f4-8997f3aeaeec","Type":"ContainerStarted","Data":"f2f4675ff158972f37f0f3bc9ac6939ab80d09b3bb6d02a85851ef429399714e"} Jul 10 19:15:32.868889 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.868844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:32.886928 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.886906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-config\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.887024 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.886943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xndp9\" (UniqueName: \"kubernetes.io/projected/1087aae5-7079-4efc-9684-9f77d24d3a5a-kube-api-access-xndp9\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.887024 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.886985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-serving-cert\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.887136 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.887059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-service-ca\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.887136 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.887114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-oauth-config\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.887243 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.887186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-oauth-serving-cert\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.887243 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.887221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-trusted-ca-bundle\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.976083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.976060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:32.988541 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.988519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-service-ca\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.988604 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.988554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-oauth-config\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.988604 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.988582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-oauth-serving-cert\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.988704 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.988618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-trusted-ca-bundle\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.988704 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.988688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-config\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.988778 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.988717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xndp9\" (UniqueName: \"kubernetes.io/projected/1087aae5-7079-4efc-9684-9f77d24d3a5a-kube-api-access-xndp9\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.988778 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.988737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-serving-cert\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.989358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.989330 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-service-ca\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.989447 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.989399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-oauth-serving-cert\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.989447 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.989442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-config\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.989565 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.989467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1087aae5-7079-4efc-9684-9f77d24d3a5a-trusted-ca-bundle\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.991004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.990985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-serving-cert\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.991089 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.991043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1087aae5-7079-4efc-9684-9f77d24d3a5a-console-oauth-config\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:32.997196 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:32.997174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xndp9\" (UniqueName: \"kubernetes.io/projected/1087aae5-7079-4efc-9684-9f77d24d3a5a-kube-api-access-xndp9\") pod \"console-6fd6dbb547-np9wr\" (UID: \"1087aae5-7079-4efc-9684-9f77d24d3a5a\") " pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:33.089301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:33.089277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:33.221074 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:33.221038 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd6dbb547-np9wr"] Jul 10 19:15:33.225187 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:15:33.225159 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1087aae5_7079_4efc_9684_9f77d24d3a5a.slice/crio-d963aee367f9643c5d8beb09e0cee0b80cb05ca5f5a727f33b1eb08f01b3c520 WatchSource:0}: Error finding container d963aee367f9643c5d8beb09e0cee0b80cb05ca5f5a727f33b1eb08f01b3c520: Status 404 returned error can't find the container with id d963aee367f9643c5d8beb09e0cee0b80cb05ca5f5a727f33b1eb08f01b3c520 Jul 10 19:15:33.835108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:33.835075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd6dbb547-np9wr" event={"ID":"1087aae5-7079-4efc-9684-9f77d24d3a5a","Type":"ContainerStarted","Data":"c8aaf80b42def73e1d6dcbe678af468ff144d6f37445fc10702bbced5bfe0add"} Jul 10 19:15:33.835278 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:33.835135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd6dbb547-np9wr" event={"ID":"1087aae5-7079-4efc-9684-9f77d24d3a5a","Type":"ContainerStarted","Data":"d963aee367f9643c5d8beb09e0cee0b80cb05ca5f5a727f33b1eb08f01b3c520"} Jul 10 19:15:33.863059 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:33.863009 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd6dbb547-np9wr" podStartSLOduration=1.8629881780000002 podStartE2EDuration="1.862988178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:15:33.861106192 +0000 UTC m=+473.899979220" watchObservedRunningTime="2026-07-10 19:15:33.862988178 +0000 UTC m=+473.901861197" Jul 10 19:15:34.844427 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:34.844379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:34.967141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:34.967105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:35.199865 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:35.199784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:35.269379 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:35.269348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:35.603048 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:35.603015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:35.708122 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:35.708084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:38.860527 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:38.860495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" event={"ID":"293eb0dd-0cdf-432d-95f4-8997f3aeaeec","Type":"ContainerStarted","Data":"5062440c316a9bbcb9d2ea23ea98ee17ddbca0424b4091b7323e0d7bcc6258f5"} Jul 10 19:15:38.878480 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:38.878431 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-9gxd9" podStartSLOduration=1.61922157 podStartE2EDuration="6.878414531s" totalImagesPullingTime="5.259192961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:15:38.877530704 +0000 UTC m=+478.916403722" watchObservedRunningTime="2026-07-10 19:15:38.878414531 +0000 UTC m=+478.917287548" Jul 10 19:15:43.089537 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:43.089493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:43.089927 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:43.089571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:43.094156 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:43.094135 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:43.450924 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:43.450837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:43.602641 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:43.602612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:43.887122 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:43.887092 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fd6dbb547-np9wr" Jul 10 19:15:44.113994 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:44.113963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:44.217567 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:44.217501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:44.314001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:44.313974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:44.417994 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:44.417968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:45.905981 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:45.905950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:46.002687 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:46.002646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:50.627084 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:50.627047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:50.758893 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:50.758859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:55.925214 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:55.925181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:56.025566 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:56.025504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:56.218673 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:56.218581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:56.319241 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:56.319210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:56.478848 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:56.478780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:56.589295 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:56.589272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:57.530355 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:57.530320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:57.593893 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:15:57.593864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:05.224922 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:05.224885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:05.324538 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:05.324503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:07.689865 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:07.689830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:07.794964 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:07.794935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:11.670305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:11.670272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:12.001968 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:12.001881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:12.455892 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:12.455860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:12.592976 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:12.592945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:14.358454 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.358423 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-shqvc"] Jul 10 19:16:14.361226 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.361200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:14.362093 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.362069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.366033 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.366001 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 19:16:14.375937 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.375918 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-shqvc"] Jul 10 19:16:14.432723 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.432696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245p7\" (UniqueName: \"kubernetes.io/projected/44cc2092-2b04-4374-a5d5-791879ba58d5-kube-api-access-245p7\") pod \"limitador-limitador-ff68ff5cb-shqvc\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.432855 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.432725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44cc2092-2b04-4374-a5d5-791879ba58d5-config-file\") pod \"limitador-limitador-ff68ff5cb-shqvc\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.436113 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.436092 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-shqvc"] Jul 10 19:16:14.489808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.489784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:14.533409 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.533384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245p7\" (UniqueName: \"kubernetes.io/projected/44cc2092-2b04-4374-a5d5-791879ba58d5-kube-api-access-245p7\") pod \"limitador-limitador-ff68ff5cb-shqvc\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.533501 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.533412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44cc2092-2b04-4374-a5d5-791879ba58d5-config-file\") pod \"limitador-limitador-ff68ff5cb-shqvc\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.534044 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.534026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44cc2092-2b04-4374-a5d5-791879ba58d5-config-file\") pod \"limitador-limitador-ff68ff5cb-shqvc\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.544018 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.544001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-245p7\" (UniqueName: \"kubernetes.io/projected/44cc2092-2b04-4374-a5d5-791879ba58d5-kube-api-access-245p7\") pod \"limitador-limitador-ff68ff5cb-shqvc\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.673027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.672963 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:14.820758 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:14.820727 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-shqvc"] Jul 10 19:16:14.821668 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:16:14.821631 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cc2092_2b04_4374_a5d5_791879ba58d5.slice/crio-8618fb6ee763bf015ba2e75f5e55a1bcfd11f7cd96a7c2782ac6db6dbcebeb15 WatchSource:0}: Error finding container 8618fb6ee763bf015ba2e75f5e55a1bcfd11f7cd96a7c2782ac6db6dbcebeb15: Status 404 returned error can't find the container with id 8618fb6ee763bf015ba2e75f5e55a1bcfd11f7cd96a7c2782ac6db6dbcebeb15 Jul 10 19:16:15.022768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:15.022684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" event={"ID":"44cc2092-2b04-4374-a5d5-791879ba58d5","Type":"ContainerStarted","Data":"8618fb6ee763bf015ba2e75f5e55a1bcfd11f7cd96a7c2782ac6db6dbcebeb15"} Jul 10 19:16:15.333921 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:15.333896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:15.463135 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:15.463110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:19.044197 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:19.044112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" event={"ID":"44cc2092-2b04-4374-a5d5-791879ba58d5","Type":"ContainerStarted","Data":"c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5"} Jul 10 19:16:19.044548 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:19.044216 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:19.061245 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:19.061200 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" podStartSLOduration=1.156850103 podStartE2EDuration="5.061187012s" totalImagesPullingTime="3.904336909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:16:19.059426522 +0000 UTC m=+519.098299540" watchObservedRunningTime="2026-07-10 19:16:19.061187012 +0000 UTC m=+519.100060029" Jul 10 19:16:30.050058 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:30.050028 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:30.925212 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:30.925185 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-shqvc"] Jul 10 19:16:30.925393 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:30.925372 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" podUID="44cc2092-2b04-4374-a5d5-791879ba58d5" containerName="limitador" containerID="cri-o://c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5" gracePeriod=30 Jul 10 19:16:31.465803 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:31.465781 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:31.482058 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:31.482025 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-245p7\" (UniqueName: \"kubernetes.io/projected/44cc2092-2b04-4374-a5d5-791879ba58d5-kube-api-access-245p7\") pod \"44cc2092-2b04-4374-a5d5-791879ba58d5\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " Jul 10 19:16:31.482177 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:31.482125 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44cc2092-2b04-4374-a5d5-791879ba58d5-config-file\") pod \"44cc2092-2b04-4374-a5d5-791879ba58d5\" (UID: \"44cc2092-2b04-4374-a5d5-791879ba58d5\") " Jul 10 19:16:31.482624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:31.482597 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cc2092-2b04-4374-a5d5-791879ba58d5-config-file" (OuterVolumeSpecName: "config-file") pod "44cc2092-2b04-4374-a5d5-791879ba58d5" (UID: "44cc2092-2b04-4374-a5d5-791879ba58d5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:16:31.484548 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:31.484498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cc2092-2b04-4374-a5d5-791879ba58d5-kube-api-access-245p7" (OuterVolumeSpecName: "kube-api-access-245p7") pod "44cc2092-2b04-4374-a5d5-791879ba58d5" (UID: "44cc2092-2b04-4374-a5d5-791879ba58d5"). InnerVolumeSpecName "kube-api-access-245p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:16:31.583015 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:31.582981 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245p7\" (UniqueName: \"kubernetes.io/projected/44cc2092-2b04-4374-a5d5-791879ba58d5-kube-api-access-245p7\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:16:31.583015 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:31.583011 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44cc2092-2b04-4374-a5d5-791879ba58d5-config-file\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:16:32.100551 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.100473 2579 generic.go:356] "Generic (PLEG): container finished" podID="44cc2092-2b04-4374-a5d5-791879ba58d5" containerID="c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5" exitCode=0 Jul 10 19:16:32.100551 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.100518 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" Jul 10 19:16:32.100551 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.100517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" event={"ID":"44cc2092-2b04-4374-a5d5-791879ba58d5","Type":"ContainerDied","Data":"c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5"} Jul 10 19:16:32.100817 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.100576 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-shqvc" event={"ID":"44cc2092-2b04-4374-a5d5-791879ba58d5","Type":"ContainerDied","Data":"8618fb6ee763bf015ba2e75f5e55a1bcfd11f7cd96a7c2782ac6db6dbcebeb15"} Jul 10 19:16:32.100817 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.100600 2579 scope.go:117] "RemoveContainer" containerID="c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5" Jul 10 19:16:32.109091 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.109071 2579 scope.go:117] "RemoveContainer" containerID="c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5" Jul 10 19:16:32.109355 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:16:32.109338 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5\": container with ID starting with c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5 not found: ID does not exist" containerID="c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5" Jul 10 19:16:32.109413 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.109362 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5"} err="failed to get container status \"c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5\": rpc error: code = NotFound desc = could not find container \"c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5\": container with ID starting with c3624ec67357d470c669f771d5f6e230210d29d1bab06f2251a3898357e11ff5 not found: ID does not exist" Jul 10 19:16:32.122485 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.122461 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-shqvc"] Jul 10 19:16:32.127815 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:32.127795 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-shqvc"] Jul 10 19:16:33.070292 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:33.070251 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cc2092-2b04-4374-a5d5-791879ba58d5" path="/var/lib/kubelet/pods/44cc2092-2b04-4374-a5d5-791879ba58d5/volumes" Jul 10 19:16:35.405547 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.405513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:35.501981 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.501958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:35.855434 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.855409 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql"] Jul 10 19:16:35.855803 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.855788 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cc2092-2b04-4374-a5d5-791879ba58d5" containerName="limitador" Jul 10 19:16:35.855858 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.855804 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cc2092-2b04-4374-a5d5-791879ba58d5" containerName="limitador" Jul 10 19:16:35.855895 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.855873 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cc2092-2b04-4374-a5d5-791879ba58d5" containerName="limitador" Jul 10 19:16:35.858809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.858789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:35.861756 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.861734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:16:35.861875 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.861739 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:16:35.861875 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.861800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 19:16:35.861875 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.861804 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4c28g\"" Jul 10 19:16:35.868782 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.868761 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql"] Jul 10 19:16:35.892950 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.892931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:35.913179 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.913160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d90297b8-8226-4aef-86f9-d3f8262c1e8c-data\") pod \"postgres-69fb4594cf-fj2ql\" (UID: \"d90297b8-8226-4aef-86f9-d3f8262c1e8c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:35.913277 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:35.913194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j69p\" (UniqueName: \"kubernetes.io/projected/d90297b8-8226-4aef-86f9-d3f8262c1e8c-kube-api-access-9j69p\") pod \"postgres-69fb4594cf-fj2ql\" (UID: \"d90297b8-8226-4aef-86f9-d3f8262c1e8c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:36.013614 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:36.013584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d90297b8-8226-4aef-86f9-d3f8262c1e8c-data\") pod \"postgres-69fb4594cf-fj2ql\" (UID: \"d90297b8-8226-4aef-86f9-d3f8262c1e8c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:36.013752 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:36.013629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j69p\" (UniqueName: \"kubernetes.io/projected/d90297b8-8226-4aef-86f9-d3f8262c1e8c-kube-api-access-9j69p\") pod \"postgres-69fb4594cf-fj2ql\" (UID: \"d90297b8-8226-4aef-86f9-d3f8262c1e8c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:36.013949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:36.013930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d90297b8-8226-4aef-86f9-d3f8262c1e8c-data\") pod \"postgres-69fb4594cf-fj2ql\" (UID: \"d90297b8-8226-4aef-86f9-d3f8262c1e8c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:36.021838 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:36.021815 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j69p\" (UniqueName: \"kubernetes.io/projected/d90297b8-8226-4aef-86f9-d3f8262c1e8c-kube-api-access-9j69p\") pod \"postgres-69fb4594cf-fj2ql\" (UID: \"d90297b8-8226-4aef-86f9-d3f8262c1e8c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:36.171140 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:36.171053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:36.305000 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:36.304874 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql"] Jul 10 19:16:36.307630 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:16:36.307600 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90297b8_8226_4aef_86f9_d3f8262c1e8c.slice/crio-7d5f9e8e7d22dd711bdee57945399c00dbcd936f7abb118ea2877873713e6dcb WatchSource:0}: Error finding container 7d5f9e8e7d22dd711bdee57945399c00dbcd936f7abb118ea2877873713e6dcb: Status 404 returned error can't find the container with id 7d5f9e8e7d22dd711bdee57945399c00dbcd936f7abb118ea2877873713e6dcb Jul 10 19:16:37.127282 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:37.127232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" event={"ID":"d90297b8-8226-4aef-86f9-d3f8262c1e8c","Type":"ContainerStarted","Data":"7d5f9e8e7d22dd711bdee57945399c00dbcd936f7abb118ea2877873713e6dcb"} Jul 10 19:16:41.221754 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:41.221733 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 19:16:42.153514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:42.153483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" event={"ID":"d90297b8-8226-4aef-86f9-d3f8262c1e8c","Type":"ContainerStarted","Data":"1cc92bdd63b701011bedab63c0e18761eccbc05d527ee352fdd2177ce8e6e776"} Jul 10 19:16:42.153699 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:42.153611 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:42.173504 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:42.173449 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" podStartSLOduration=2.263745038 podStartE2EDuration="7.173436029s" totalImagesPullingTime="4.909690991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:16:42.171808076 +0000 UTC m=+542.210681093" watchObservedRunningTime="2026-07-10 19:16:42.173436029 +0000 UTC m=+542.212309045" Jul 10 19:16:48.187737 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:48.187705 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fj2ql" Jul 10 19:16:48.489050 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:48.488978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:48.626808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:48.626780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:54.736541 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:54.736511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:54.826480 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:54.826449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:54.901697 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:54.901648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:57.697866 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:57.697831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:57.796368 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:16:57.796339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:01.788247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:01.788216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:27.136427 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.136397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:27.692865 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.692830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:27.739586 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.739554 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5"] Jul 10 19:17:27.742324 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.742309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.744976 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.744956 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jxtcn\"" Jul 10 19:17:27.745052 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.745015 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 19:17:27.751534 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.751494 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5"] Jul 10 19:17:27.831808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.831782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ba7900-8012-4776-b19c-ae1fc1ec52dc-maas-api-tls\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.831919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.831810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knftc\" (UniqueName: \"kubernetes.io/projected/82ba7900-8012-4776-b19c-ae1fc1ec52dc-kube-api-access-knftc\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.831919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.831847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ba7900-8012-4776-b19c-ae1fc1ec52dc-openshift-service-ca\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.933066 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.933039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ba7900-8012-4776-b19c-ae1fc1ec52dc-maas-api-tls\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.933205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.933070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knftc\" (UniqueName: \"kubernetes.io/projected/82ba7900-8012-4776-b19c-ae1fc1ec52dc-kube-api-access-knftc\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.933205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.933110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ba7900-8012-4776-b19c-ae1fc1ec52dc-openshift-service-ca\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.933205 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:17:27.933195 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 10 19:17:27.933330 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:17:27.933263 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ba7900-8012-4776-b19c-ae1fc1ec52dc-maas-api-tls podName:82ba7900-8012-4776-b19c-ae1fc1ec52dc nodeName:}" failed. No retries permitted until 2026-07-10 19:17:28.433248191 +0000 UTC m=+588.472121191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82ba7900-8012-4776-b19c-ae1fc1ec52dc-maas-api-tls") pod "maas-api-6b4cfccc4d-kb4m5" (UID: "82ba7900-8012-4776-b19c-ae1fc1ec52dc") : secret "maas-api-serving-cert" not found Jul 10 19:17:27.933784 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.933766 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ba7900-8012-4776-b19c-ae1fc1ec52dc-openshift-service-ca\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:27.941785 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:27.941762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knftc\" (UniqueName: \"kubernetes.io/projected/82ba7900-8012-4776-b19c-ae1fc1ec52dc-kube-api-access-knftc\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:28.119001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.118970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:28.219969 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.219939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:28.295556 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.295515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:28.296242 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.296220 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-gmpsm"] Jul 10 19:17:28.298569 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.298548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.301207 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.301188 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6wkvj\"" Jul 10 19:17:28.301323 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.301205 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 19:17:28.307100 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.307075 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-gmpsm"] Jul 10 19:17:28.438834 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.438771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5kj\" (UniqueName: \"kubernetes.io/projected/b6a19137-9eac-4215-a5e2-d8c38972c332-kube-api-access-bk5kj\") pod \"payload-processing-58f86c7798-gmpsm\" (UID: \"b6a19137-9eac-4215-a5e2-d8c38972c332\") " pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.438834 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.438813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ba7900-8012-4776-b19c-ae1fc1ec52dc-maas-api-tls\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:28.438984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.438916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a19137-9eac-4215-a5e2-d8c38972c332-plugins-config-volume\") pod \"payload-processing-58f86c7798-gmpsm\" (UID: \"b6a19137-9eac-4215-a5e2-d8c38972c332\") " pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.441067 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.441041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ba7900-8012-4776-b19c-ae1fc1ec52dc-maas-api-tls\") pod \"maas-api-6b4cfccc4d-kb4m5\" (UID: \"82ba7900-8012-4776-b19c-ae1fc1ec52dc\") " pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:28.540198 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.540170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5kj\" (UniqueName: \"kubernetes.io/projected/b6a19137-9eac-4215-a5e2-d8c38972c332-kube-api-access-bk5kj\") pod \"payload-processing-58f86c7798-gmpsm\" (UID: \"b6a19137-9eac-4215-a5e2-d8c38972c332\") " pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.540313 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.540272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a19137-9eac-4215-a5e2-d8c38972c332-plugins-config-volume\") pod \"payload-processing-58f86c7798-gmpsm\" (UID: \"b6a19137-9eac-4215-a5e2-d8c38972c332\") " pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.540959 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.540942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a19137-9eac-4215-a5e2-d8c38972c332-plugins-config-volume\") pod \"payload-processing-58f86c7798-gmpsm\" (UID: \"b6a19137-9eac-4215-a5e2-d8c38972c332\") " pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.547956 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.547930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5kj\" (UniqueName: \"kubernetes.io/projected/b6a19137-9eac-4215-a5e2-d8c38972c332-kube-api-access-bk5kj\") pod \"payload-processing-58f86c7798-gmpsm\" (UID: \"b6a19137-9eac-4215-a5e2-d8c38972c332\") " pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.609382 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.609359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:28.653720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.653695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:28.751240 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.751212 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-gmpsm"] Jul 10 19:17:28.752235 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:17:28.752199 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a19137_9eac_4215_a5e2_d8c38972c332.slice/crio-caa7375db4a4d69b411994c96a5f3fa10a5f2750956a62cb58ad784c8234d476 WatchSource:0}: Error finding container caa7375db4a4d69b411994c96a5f3fa10a5f2750956a62cb58ad784c8234d476: Status 404 returned error can't find the container with id caa7375db4a4d69b411994c96a5f3fa10a5f2750956a62cb58ad784c8234d476 Jul 10 19:17:28.789608 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:28.789583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5"] Jul 10 19:17:28.791339 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:17:28.791314 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ba7900_8012_4776_b19c_ae1fc1ec52dc.slice/crio-83dfddc0ba3369bd162c3fcd80e81cfe32752a4712e710c0a89cd43a941cb5da WatchSource:0}: Error finding container 83dfddc0ba3369bd162c3fcd80e81cfe32752a4712e710c0a89cd43a941cb5da: Status 404 returned error can't find the container with id 83dfddc0ba3369bd162c3fcd80e81cfe32752a4712e710c0a89cd43a941cb5da Jul 10 19:17:29.360152 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:29.360120 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" event={"ID":"82ba7900-8012-4776-b19c-ae1fc1ec52dc","Type":"ContainerStarted","Data":"83dfddc0ba3369bd162c3fcd80e81cfe32752a4712e710c0a89cd43a941cb5da"} Jul 10 19:17:29.361235 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:29.361214 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" event={"ID":"b6a19137-9eac-4215-a5e2-d8c38972c332","Type":"ContainerStarted","Data":"caa7375db4a4d69b411994c96a5f3fa10a5f2750956a62cb58ad784c8234d476"} Jul 10 19:17:30.369544 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:30.369509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" event={"ID":"b6a19137-9eac-4215-a5e2-d8c38972c332","Type":"ContainerStarted","Data":"b06652db3da03d2fb412e20291fbf82dea78071006bc9b9b31d037f50e542e20"} Jul 10 19:17:30.370002 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:30.369617 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:17:30.388755 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:30.388691 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" podStartSLOduration=0.933205118 podStartE2EDuration="2.388666647s" totalImagesPullingTime="1.455461529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:17:30.385078841 +0000 UTC m=+590.423951859" watchObservedRunningTime="2026-07-10 19:17:30.388666647 +0000 UTC m=+590.427539655" Jul 10 19:17:32.381247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:32.381212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" event={"ID":"82ba7900-8012-4776-b19c-ae1fc1ec52dc","Type":"ContainerStarted","Data":"10cd6c0f4418f544a32abeb74ae2006fa3591732c205637c0c222f938580a33a"} Jul 10 19:17:32.381643 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:32.381290 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:32.399224 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:32.399185 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" podStartSLOduration=2.737356564 podStartE2EDuration="5.399171034s" totalImagesPullingTime="2.66181447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:17:32.397398071 +0000 UTC m=+592.436271100" watchObservedRunningTime="2026-07-10 19:17:32.399171034 +0000 UTC m=+592.438044050" Jul 10 19:17:38.391228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:38.391202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b4cfccc4d-kb4m5" Jul 10 19:17:39.350320 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:39.350289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:39.446642 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:39.446606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:40.875403 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:40.875379 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:17:40.876132 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:40.876105 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:17:41.377311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:17:41.377282 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-gmpsm" Jul 10 19:18:06.411731 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:06.411664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:18:25.074628 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.074596 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm"] Jul 10 19:18:25.077049 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.077033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.079810 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.079790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mmzcw\"" Jul 10 19:18:25.079919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.079789 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 19:18:25.081052 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.081036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 19:18:25.081320 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.081307 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 19:18:25.087761 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.087741 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm"] Jul 10 19:18:25.106044 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.106008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:18:25.202328 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.202305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.202433 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.202354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl6n8\" (UniqueName: \"kubernetes.io/projected/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-kube-api-access-nl6n8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.202504 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.202462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.202504 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.202495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.202622 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.202530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.202622 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.202544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303366 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303551 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303551 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl6n8\" (UniqueName: \"kubernetes.io/projected/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-kube-api-access-nl6n8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303637 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303868 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303955 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.303955 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.303915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.305591 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.305561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.305851 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.305834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.312925 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.312883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl6n8\" (UniqueName: \"kubernetes.io/projected/0e7a28d7-af9f-4d9e-9b78-d3e542d021e3-kube-api-access-nl6n8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2dlfm\" (UID: \"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.388341 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.388309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:25.514537 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.514513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm"] Jul 10 19:18:25.516243 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:18:25.516213 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7a28d7_af9f_4d9e_9b78_d3e542d021e3.slice/crio-1fb145673a7eca2240c8b49744d896bb4055a70c5cb809ec1d99d47d6d73e9b7 WatchSource:0}: Error finding container 1fb145673a7eca2240c8b49744d896bb4055a70c5cb809ec1d99d47d6d73e9b7: Status 404 returned error can't find the container with id 1fb145673a7eca2240c8b49744d896bb4055a70c5cb809ec1d99d47d6d73e9b7 Jul 10 19:18:25.616992 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:25.616969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" event={"ID":"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3","Type":"ContainerStarted","Data":"1fb145673a7eca2240c8b49744d896bb4055a70c5cb809ec1d99d47d6d73e9b7"} Jul 10 19:18:26.623442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:26.623409 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" event={"ID":"0e7a28d7-af9f-4d9e-9b78-d3e542d021e3","Type":"ContainerStarted","Data":"bf11862b089236cf8f15c4096fda0c25f100da927e5de2e3d1384393161aeb68"} Jul 10 19:18:26.646243 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:26.646197 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" podStartSLOduration=0.682781296 podStartE2EDuration="1.646181962s" totalImagesPullingTime="963.400666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:18:26.644924396 +0000 UTC m=+646.683797414" watchObservedRunningTime="2026-07-10 19:18:26.646181962 +0000 UTC m=+646.685054978" Jul 10 19:18:35.389192 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:35.389158 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:35.389638 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:35.389217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:35.401863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:35.401841 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:35.674968 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:35.674895 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2dlfm" Jul 10 19:18:55.589108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:18:55.589074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:19:10.466036 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.466003 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr"] Jul 10 19:19:10.469455 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.469439 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.472396 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.472378 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 19:19:10.479372 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.479345 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr"] Jul 10 19:19:10.562897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.562867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:19:10.575996 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.575969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26sm\" (UniqueName: \"kubernetes.io/projected/48346b08-fc9c-4e8a-9c54-e1f106b0e585-kube-api-access-r26sm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.576085 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.576007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.576085 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.576032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.576162 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.576090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.576162 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.576124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346b08-fc9c-4e8a-9c54-e1f106b0e585-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.576225 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.576168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677287 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26sm\" (UniqueName: \"kubernetes.io/projected/48346b08-fc9c-4e8a-9c54-e1f106b0e585-kube-api-access-r26sm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677381 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346b08-fc9c-4e8a-9c54-e1f106b0e585-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677536 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677775 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677893 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677877 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.677933 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.677872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.679636 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.679620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48346b08-fc9c-4e8a-9c54-e1f106b0e585-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.679922 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.679904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48346b08-fc9c-4e8a-9c54-e1f106b0e585-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.685581 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.685554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26sm\" (UniqueName: \"kubernetes.io/projected/48346b08-fc9c-4e8a-9c54-e1f106b0e585-kube-api-access-r26sm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr\" (UID: \"48346b08-fc9c-4e8a-9c54-e1f106b0e585\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.780370 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.780309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:10.908556 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.908530 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr"] Jul 10 19:19:10.910389 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:19:10.910361 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48346b08_fc9c_4e8a_9c54_e1f106b0e585.slice/crio-0afdb70a36e2e478e05bd0536f419de789c880362feac1a9e8b7ed277cd4de91 WatchSource:0}: Error finding container 0afdb70a36e2e478e05bd0536f419de789c880362feac1a9e8b7ed277cd4de91: Status 404 returned error can't find the container with id 0afdb70a36e2e478e05bd0536f419de789c880362feac1a9e8b7ed277cd4de91 Jul 10 19:19:10.911730 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:10.911710 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:19:11.828949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:11.828917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" event={"ID":"48346b08-fc9c-4e8a-9c54-e1f106b0e585","Type":"ContainerStarted","Data":"efde3731a7654afeed5106abbc54097c0e2f0c07206bb09604f81055c68be40a"} Jul 10 19:19:11.829280 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:11.828965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" event={"ID":"48346b08-fc9c-4e8a-9c54-e1f106b0e585","Type":"ContainerStarted","Data":"0afdb70a36e2e478e05bd0536f419de789c880362feac1a9e8b7ed277cd4de91"} Jul 10 19:19:11.860424 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:11.860381 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" podStartSLOduration=1.686815231 podStartE2EDuration="1.86036719s" totalImagesPullingTime="173.551959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:11.85907908 +0000 UTC m=+691.897952212" watchObservedRunningTime="2026-07-10 19:19:11.86036719 +0000 UTC m=+691.899240206" Jul 10 19:19:20.781195 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:20.781161 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:20.781640 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:20.781219 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:20.793522 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:20.793500 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:20.886461 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:20.886441 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr" Jul 10 19:19:23.920341 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:23.920306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:19:23.971995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:23.971968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:19:26.711575 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:26.711527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:19:39.866353 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:19:39.866324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:00.023582 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:20:00.023552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:33.293160 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:20:33.293102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:33.392500 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:20:33.392472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:34.397699 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:20:34.397648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:34.489637 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:20:34.489609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:54.895522 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:20:54.895485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:21:44.232632 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:21:44.232594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:21:45.859490 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:21:45.859456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:16.565383 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:16.565346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:16.622791 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:16.622763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:26.514538 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:26.514497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:26.612224 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:26.612191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:26.717204 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:26.717177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:32.468434 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:32.468341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:32.569236 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:32.569200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:32.668603 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:32.668575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:32.796092 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:32.796000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:34.182578 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:34.182525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:35.889116 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:35.889076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:35.990039 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:35.990008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:38.642143 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:38.642110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:38.739370 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:38.739320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:22:40.908097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:40.908065 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:22:40.910957 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:22:40.910937 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:27:40.942490 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:27:40.942462 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:27:40.945601 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:27:40.945581 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:27:56.424067 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:27:56.424036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:27.871350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:30:27.871273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:00.045335 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:31:00.045298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:03.774843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:32:03.774810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:40.980091 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:32:40.980065 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:32:40.982985 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:32:40.982967 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:33:10.423512 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:10.423427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:16.874409 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:16.874368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:18.280488 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:18.280447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:25.125413 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:25.125373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:30.168377 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:30.168344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:32.920478 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:32.920447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:33.021103 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:33.021071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:33.366141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:33.366109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:39.219408 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:39.219369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:39.322014 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:39.321986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:39.445554 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:39.445506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:39.542912 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:39.542837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:40.625948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.625900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:40.727867 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.727841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:40.751584 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.751548 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd"] Jul 10 19:33:40.755129 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.755102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.757792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.757770 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f5c7829e-serving-cert\"" Jul 10 19:33:40.763008 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.762925 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd"] Jul 10 19:33:40.795947 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.795921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4dtv\" (UniqueName: \"kubernetes.io/projected/63e0a050-197f-41f7-b295-2d016b638308-kube-api-access-f4dtv\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.796097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.795960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.796097 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.795980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0a050-197f-41f7-b295-2d016b638308-openshift-service-ca\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.824326 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.824301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:40.896946 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.896869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4dtv\" (UniqueName: \"kubernetes.io/projected/63e0a050-197f-41f7-b295-2d016b638308-kube-api-access-f4dtv\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.896946 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.896926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.897104 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.896958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0a050-197f-41f7-b295-2d016b638308-openshift-service-ca\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.897104 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:33:40.897063 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-serving-cert: secret "maas-api-e2e-ait-f5c7829e-serving-cert" not found Jul 10 19:33:40.897208 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:33:40.897141 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls podName:63e0a050-197f-41f7-b295-2d016b638308 nodeName:}" failed. No retries permitted until 2026-07-10 19:33:41.397122369 +0000 UTC m=+1561.435995374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls") pod "maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" (UID: "63e0a050-197f-41f7-b295-2d016b638308") : secret "maas-api-e2e-ait-f5c7829e-serving-cert" not found Jul 10 19:33:40.897623 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.897603 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0a050-197f-41f7-b295-2d016b638308-openshift-service-ca\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:40.908184 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:40.908162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4dtv\" (UniqueName: \"kubernetes.io/projected/63e0a050-197f-41f7-b295-2d016b638308-kube-api-access-f4dtv\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:41.134122 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.134086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:41.282053 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.281970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:41.382006 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.381975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:41.400622 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.400599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:41.402853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.402829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls\") pod \"maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:41.667407 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.667382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:41.796310 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.796177 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd"] Jul 10 19:33:41.798805 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:33:41.798776 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e0a050_197f_41f7_b295_2d016b638308.slice/crio-6fcd3e7a94a23f20d3e4921376f2a2e503a49d0621d35a2228f4c5805fc0028e WatchSource:0}: Error finding container 6fcd3e7a94a23f20d3e4921376f2a2e503a49d0621d35a2228f4c5805fc0028e: Status 404 returned error can't find the container with id 6fcd3e7a94a23f20d3e4921376f2a2e503a49d0621d35a2228f4c5805fc0028e Jul 10 19:33:41.800095 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:41.800078 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:33:42.794546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:42.794473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" event={"ID":"63e0a050-197f-41f7-b295-2d016b638308","Type":"ContainerStarted","Data":"37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761"} Jul 10 19:33:42.794546 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:42.794523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" event={"ID":"63e0a050-197f-41f7-b295-2d016b638308","Type":"ContainerStarted","Data":"6fcd3e7a94a23f20d3e4921376f2a2e503a49d0621d35a2228f4c5805fc0028e"} Jul 10 19:33:42.794941 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:42.794555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:42.819059 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:42.819017 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" podStartSLOduration=2.151410625 podStartE2EDuration="2.819002178s" totalImagesPullingTime="667.591553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:33:42.81691483 +0000 UTC m=+1562.855787847" watchObservedRunningTime="2026-07-10 19:33:42.819002178 +0000 UTC m=+1562.857875196" Jul 10 19:33:43.602976 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:43.602941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:45.564980 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:45.564954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:45.761111 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:45.761079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:45.840836 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:45.840811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:46.006056 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.006025 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd"] Jul 10 19:33:46.006292 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.006262 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" podUID="63e0a050-197f-41f7-b295-2d016b638308" containerName="maas-api" containerID="cri-o://37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761" gracePeriod=30 Jul 10 19:33:46.031895 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.031874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:46.258194 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.258168 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:46.342473 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.342449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4dtv\" (UniqueName: \"kubernetes.io/projected/63e0a050-197f-41f7-b295-2d016b638308-kube-api-access-f4dtv\") pod \"63e0a050-197f-41f7-b295-2d016b638308\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " Jul 10 19:33:46.342599 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.342538 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls\") pod \"63e0a050-197f-41f7-b295-2d016b638308\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " Jul 10 19:33:46.342599 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.342591 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0a050-197f-41f7-b295-2d016b638308-openshift-service-ca\") pod \"63e0a050-197f-41f7-b295-2d016b638308\" (UID: \"63e0a050-197f-41f7-b295-2d016b638308\") " Jul 10 19:33:46.342984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.342958 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e0a050-197f-41f7-b295-2d016b638308-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63e0a050-197f-41f7-b295-2d016b638308" (UID: "63e0a050-197f-41f7-b295-2d016b638308"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:33:46.344539 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.344512 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e0a050-197f-41f7-b295-2d016b638308-kube-api-access-f4dtv" (OuterVolumeSpecName: "kube-api-access-f4dtv") pod "63e0a050-197f-41f7-b295-2d016b638308" (UID: "63e0a050-197f-41f7-b295-2d016b638308"). InnerVolumeSpecName "kube-api-access-f4dtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:46.344624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.344585 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63e0a050-197f-41f7-b295-2d016b638308" (UID: "63e0a050-197f-41f7-b295-2d016b638308"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:33:46.443930 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.443869 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63e0a050-197f-41f7-b295-2d016b638308-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:33:46.443930 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.443893 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e0a050-197f-41f7-b295-2d016b638308-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:33:46.443930 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.443905 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4dtv\" (UniqueName: \"kubernetes.io/projected/63e0a050-197f-41f7-b295-2d016b638308-kube-api-access-f4dtv\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:33:46.813939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.813857 2579 generic.go:356] "Generic (PLEG): container finished" podID="63e0a050-197f-41f7-b295-2d016b638308" containerID="37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761" exitCode=0 Jul 10 19:33:46.813939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.813901 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" Jul 10 19:33:46.813939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.813925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" event={"ID":"63e0a050-197f-41f7-b295-2d016b638308","Type":"ContainerDied","Data":"37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761"} Jul 10 19:33:46.814398 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.813964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd" event={"ID":"63e0a050-197f-41f7-b295-2d016b638308","Type":"ContainerDied","Data":"6fcd3e7a94a23f20d3e4921376f2a2e503a49d0621d35a2228f4c5805fc0028e"} Jul 10 19:33:46.814398 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.813986 2579 scope.go:117] "RemoveContainer" containerID="37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761" Jul 10 19:33:46.823333 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.823316 2579 scope.go:117] "RemoveContainer" containerID="37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761" Jul 10 19:33:46.823596 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:33:46.823575 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761\": container with ID starting with 37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761 not found: ID does not exist" containerID="37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761" Jul 10 19:33:46.823649 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.823606 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761"} err="failed to get container status \"37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761\": rpc error: code = NotFound desc = could not find container \"37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761\": container with ID starting with 37e831c2e4617f0c34f2e1a995e2e851fdff083feff24e9cbdc2d174f72ba761 not found: ID does not exist" Jul 10 19:33:46.838720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.838693 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd"] Jul 10 19:33:46.843076 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:46.843057 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5c7829e-68cc5c7776-nzqjd"] Jul 10 19:33:47.071494 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:47.071372 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e0a050-197f-41f7-b295-2d016b638308" path="/var/lib/kubelet/pods/63e0a050-197f-41f7-b295-2d016b638308/volumes" Jul 10 19:33:47.760386 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:47.760360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:51.403350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:51.403320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:51.503834 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:51.503803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:51.603697 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:51.603642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:51.699842 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:51.699777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:52.698141 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.698104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:52.777779 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.777748 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl"] Jul 10 19:33:52.778349 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.778332 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e0a050-197f-41f7-b295-2d016b638308" containerName="maas-api" Jul 10 19:33:52.778396 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.778355 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e0a050-197f-41f7-b295-2d016b638308" containerName="maas-api" Jul 10 19:33:52.778495 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.778475 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e0a050-197f-41f7-b295-2d016b638308" containerName="maas-api" Jul 10 19:33:52.783282 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.783264 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:52.786612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.786590 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0f19f196-serving-cert\"" Jul 10 19:33:52.787931 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.787908 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl"] Jul 10 19:33:52.792360 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.792260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:52.892855 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.892826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:52.898063 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.898042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-maas-api-tls\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:52.898130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.898093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-openshift-service-ca\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:52.898168 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.898140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd48p\" (UniqueName: \"kubernetes.io/projected/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-kube-api-access-qd48p\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:52.998903 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.998838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd48p\" (UniqueName: \"kubernetes.io/projected/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-kube-api-access-qd48p\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:52.999009 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.998913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-maas-api-tls\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:52.999009 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.998950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-openshift-service-ca\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:52.999479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:52.999458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-openshift-service-ca\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:53.001184 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.001163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-maas-api-tls\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:53.007821 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.007784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:53.009563 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.009544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd48p\" (UniqueName: \"kubernetes.io/projected/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-kube-api-access-qd48p\") pod \"maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:53.096251 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.096229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:53.226228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.226205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl"] Jul 10 19:33:53.228240 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:33:53.228208 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ce3471_a81d_4bb0_b3f3_a53b4736bcb9.slice/crio-868662b468658db17f98b301792d0be6dcbe677d184e9233625c5a0200553a35 WatchSource:0}: Error finding container 868662b468658db17f98b301792d0be6dcbe677d184e9233625c5a0200553a35: Status 404 returned error can't find the container with id 868662b468658db17f98b301792d0be6dcbe677d184e9233625c5a0200553a35 Jul 10 19:33:53.330320 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.330264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:53.427189 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.427161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:53.849721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.849684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" event={"ID":"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9","Type":"ContainerStarted","Data":"2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0"} Jul 10 19:33:53.850072 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.849740 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" event={"ID":"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9","Type":"ContainerStarted","Data":"868662b468658db17f98b301792d0be6dcbe677d184e9233625c5a0200553a35"} Jul 10 19:33:53.850072 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.849775 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:53.867914 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:53.867871 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" podStartSLOduration=1.452006952 podStartE2EDuration="1.867858108s" totalImagesPullingTime="415.851156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:33:53.866942891 +0000 UTC m=+1573.905815930" watchObservedRunningTime="2026-07-10 19:33:53.867858108 +0000 UTC m=+1573.906731155" Jul 10 19:33:55.195201 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:55.195162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:57.753995 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:57.753963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:58.081988 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.081910 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl"] Jul 10 19:33:58.082138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.082096 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" podUID="a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" containerName="maas-api" containerID="cri-o://2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0" gracePeriod=30 Jul 10 19:33:58.087612 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.087592 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:58.121901 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.121877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:58.325373 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.325349 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:58.450183 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.450155 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-maas-api-tls\") pod \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " Jul 10 19:33:58.450317 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.450214 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd48p\" (UniqueName: \"kubernetes.io/projected/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-kube-api-access-qd48p\") pod \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " Jul 10 19:33:58.450317 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.450294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-openshift-service-ca\") pod \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\" (UID: \"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9\") " Jul 10 19:33:58.450700 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.450675 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" (UID: "a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:33:58.452299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.452269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-kube-api-access-qd48p" (OuterVolumeSpecName: "kube-api-access-qd48p") pod "a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" (UID: "a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9"). InnerVolumeSpecName "kube-api-access-qd48p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:58.452299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.452291 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" (UID: "a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:33:58.551476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.551455 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:33:58.551476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.551476 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:33:58.551604 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.551485 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd48p\" (UniqueName: \"kubernetes.io/projected/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9-kube-api-access-qd48p\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:33:58.874850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.874819 2579 generic.go:356] "Generic (PLEG): container finished" podID="a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" containerID="2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0" exitCode=0 Jul 10 19:33:58.875208 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.874866 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" Jul 10 19:33:58.875208 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.874902 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" event={"ID":"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9","Type":"ContainerDied","Data":"2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0"} Jul 10 19:33:58.875208 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.874956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl" event={"ID":"a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9","Type":"ContainerDied","Data":"868662b468658db17f98b301792d0be6dcbe677d184e9233625c5a0200553a35"} Jul 10 19:33:58.875208 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.874978 2579 scope.go:117] "RemoveContainer" containerID="2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0" Jul 10 19:33:58.883647 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.883630 2579 scope.go:117] "RemoveContainer" containerID="2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0" Jul 10 19:33:58.883939 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:33:58.883918 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0\": container with ID starting with 2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0 not found: ID does not exist" containerID="2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0" Jul 10 19:33:58.884010 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.883953 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0"} err="failed to get container status \"2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0\": rpc error: code = NotFound desc = could not find container \"2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0\": container with ID starting with 2b69731b7eabaf4d7f4feda0f7e2594f0fe45c9790823b4072b6bf04662ed1a0 not found: ID does not exist" Jul 10 19:33:58.896099 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.896073 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl"] Jul 10 19:33:58.899916 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:58.899895 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0f19f196-59865fd56c-lxdsl"] Jul 10 19:33:59.071220 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:33:59.071190 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" path="/var/lib/kubelet/pods/a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9/volumes" Jul 10 19:34:03.815256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:03.815222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:03.915699 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:03.915646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:06.187782 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:06.187755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:09.692576 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:09.692542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:09.797011 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:09.796980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:09.896379 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:09.896349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:09.993561 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:09.993491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:10.984952 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:10.984922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.085901 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.085870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.173059 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.173029 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k"] Jul 10 19:34:11.173408 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.173393 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" containerName="maas-api" Jul 10 19:34:11.173408 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.173408 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" containerName="maas-api" Jul 10 19:34:11.173532 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.173475 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ce3471-a81d-4bb0-b3f3-a53b4736bcb9" containerName="maas-api" Jul 10 19:34:11.176525 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.176505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.179107 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.179089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-57caaf74-serving-cert\"" Jul 10 19:34:11.185920 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.185898 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k"] Jul 10 19:34:11.208913 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.208887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.258678 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.258592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2gd\" (UniqueName: \"kubernetes.io/projected/cd140e1f-18dc-4870-be14-7d74e9b9ca23-kube-api-access-nw2gd\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.258775 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.258680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.258775 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.258734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd140e1f-18dc-4870-be14-7d74e9b9ca23-openshift-service-ca\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.359721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.359688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2gd\" (UniqueName: \"kubernetes.io/projected/cd140e1f-18dc-4870-be14-7d74e9b9ca23-kube-api-access-nw2gd\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.359830 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.359755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.359830 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.359794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd140e1f-18dc-4870-be14-7d74e9b9ca23-openshift-service-ca\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.359914 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:34:11.359893 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-serving-cert: secret "maas-api-e2e-derive-57caaf74-serving-cert" not found Jul 10 19:34:11.359979 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:34:11.359969 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls podName:cd140e1f-18dc-4870-be14-7d74e9b9ca23 nodeName:}" failed. No retries permitted until 2026-07-10 19:34:11.859948849 +0000 UTC m=+1591.898821846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls") pod "maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" (UID: "cd140e1f-18dc-4870-be14-7d74e9b9ca23") : secret "maas-api-e2e-derive-57caaf74-serving-cert" not found Jul 10 19:34:11.360383 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.360364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd140e1f-18dc-4870-be14-7d74e9b9ca23-openshift-service-ca\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.374779 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.374759 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2gd\" (UniqueName: \"kubernetes.io/projected/cd140e1f-18dc-4870-be14-7d74e9b9ca23-kube-api-access-nw2gd\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.419567 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.419534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.756031 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.755998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.857392 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.857358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.861671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.861631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:11.864110 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:11.864082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls\") pod \"maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:12.087424 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:12.087404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:12.247786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:12.247756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k"] Jul 10 19:34:12.250030 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:34:12.250005 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd140e1f_18dc_4870_be14_7d74e9b9ca23.slice/crio-7a0ea132de7fd1dfeb696592d56f27bce76d5f37dc22ef712a7a1bfeb8506fed WatchSource:0}: Error finding container 7a0ea132de7fd1dfeb696592d56f27bce76d5f37dc22ef712a7a1bfeb8506fed: Status 404 returned error can't find the container with id 7a0ea132de7fd1dfeb696592d56f27bce76d5f37dc22ef712a7a1bfeb8506fed Jul 10 19:34:12.947050 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:12.946965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" event={"ID":"cd140e1f-18dc-4870-be14-7d74e9b9ca23","Type":"ContainerStarted","Data":"53ff1005a960c6dc26557e61d000b72187837b011bdb3dc279f8c87b79e0e4c9"} Jul 10 19:34:12.947050 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:12.947017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" event={"ID":"cd140e1f-18dc-4870-be14-7d74e9b9ca23","Type":"ContainerStarted","Data":"7a0ea132de7fd1dfeb696592d56f27bce76d5f37dc22ef712a7a1bfeb8506fed"} Jul 10 19:34:12.947050 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:12.947042 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:13.340619 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:13.340594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:15.399514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.399478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:15.691805 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.691728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:15.753276 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.753248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:15.832748 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.832689 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" podStartSLOduration=4.443834026 podStartE2EDuration="4.832667031s" totalImagesPullingTime="388.833005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:12.963536614 +0000 UTC m=+1593.002409631" watchObservedRunningTime="2026-07-10 19:34:15.832667031 +0000 UTC m=+1595.871540042" Jul 10 19:34:15.835861 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.835837 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k"] Jul 10 19:34:15.836244 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.836216 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" podUID="cd140e1f-18dc-4870-be14-7d74e9b9ca23" containerName="maas-api" containerID="cri-o://53ff1005a960c6dc26557e61d000b72187837b011bdb3dc279f8c87b79e0e4c9" gracePeriod=30 Jul 10 19:34:15.868202 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.868181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:15.963526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.963503 2579 generic.go:356] "Generic (PLEG): container finished" podID="cd140e1f-18dc-4870-be14-7d74e9b9ca23" containerID="53ff1005a960c6dc26557e61d000b72187837b011bdb3dc279f8c87b79e0e4c9" exitCode=0 Jul 10 19:34:15.963633 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:15.963578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" event={"ID":"cd140e1f-18dc-4870-be14-7d74e9b9ca23","Type":"ContainerDied","Data":"53ff1005a960c6dc26557e61d000b72187837b011bdb3dc279f8c87b79e0e4c9"} Jul 10 19:34:16.087934 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.087906 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:16.091481 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.091462 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls\") pod \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " Jul 10 19:34:16.091614 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.091517 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw2gd\" (UniqueName: \"kubernetes.io/projected/cd140e1f-18dc-4870-be14-7d74e9b9ca23-kube-api-access-nw2gd\") pod \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " Jul 10 19:34:16.091614 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.091577 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd140e1f-18dc-4870-be14-7d74e9b9ca23-openshift-service-ca\") pod \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\" (UID: \"cd140e1f-18dc-4870-be14-7d74e9b9ca23\") " Jul 10 19:34:16.091984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.091958 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd140e1f-18dc-4870-be14-7d74e9b9ca23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd140e1f-18dc-4870-be14-7d74e9b9ca23" (UID: "cd140e1f-18dc-4870-be14-7d74e9b9ca23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:34:16.093535 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.093516 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd140e1f-18dc-4870-be14-7d74e9b9ca23" (UID: "cd140e1f-18dc-4870-be14-7d74e9b9ca23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:34:16.093535 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.093515 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd140e1f-18dc-4870-be14-7d74e9b9ca23-kube-api-access-nw2gd" (OuterVolumeSpecName: "kube-api-access-nw2gd") pod "cd140e1f-18dc-4870-be14-7d74e9b9ca23" (UID: "cd140e1f-18dc-4870-be14-7d74e9b9ca23"). InnerVolumeSpecName "kube-api-access-nw2gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:34:16.192834 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.192806 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd140e1f-18dc-4870-be14-7d74e9b9ca23-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:34:16.192834 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.192834 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2gd\" (UniqueName: \"kubernetes.io/projected/cd140e1f-18dc-4870-be14-7d74e9b9ca23-kube-api-access-nw2gd\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:34:16.193050 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.192846 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd140e1f-18dc-4870-be14-7d74e9b9ca23-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:34:16.969436 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.969400 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" Jul 10 19:34:16.969436 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.969421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k" event={"ID":"cd140e1f-18dc-4870-be14-7d74e9b9ca23","Type":"ContainerDied","Data":"7a0ea132de7fd1dfeb696592d56f27bce76d5f37dc22ef712a7a1bfeb8506fed"} Jul 10 19:34:16.969948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.969475 2579 scope.go:117] "RemoveContainer" containerID="53ff1005a960c6dc26557e61d000b72187837b011bdb3dc279f8c87b79e0e4c9" Jul 10 19:34:16.991047 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.991022 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k"] Jul 10 19:34:16.994093 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:16.994072 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57caaf74-5b8f47c6cb-4s55k"] Jul 10 19:34:17.071434 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:17.071412 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd140e1f-18dc-4870-be14-7d74e9b9ca23" path="/var/lib/kubelet/pods/cd140e1f-18dc-4870-be14-7d74e9b9ca23/volumes" Jul 10 19:34:17.346840 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:17.346806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:21.367178 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:21.367147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:21.467123 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:21.467094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:21.810458 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:21.810386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:21.909296 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:21.909270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:22.008923 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:22.008899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:23.343318 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:23.343286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.137055 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:29.137020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.193844 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:29.193810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.272079 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:29.272053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.372170 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:29.372133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.475111 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:29.475041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.800896 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:29.800806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:55.717081 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:34:55.717047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:01.530464 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:01.530434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:01.628262 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:01.628237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:01.729242 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:01.729213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:09.953423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:09.953392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:10.002036 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:10.002013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:17.516368 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:17.516334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:23.391553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:23.391522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:23.503001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:23.502971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:23.591768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:23.591741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:34.138811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:34.138768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:34.876119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:34.876087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:40.437807 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:40.437748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:40.498509 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:40.498478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:40.591722 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:40.591695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:40.688171 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:40.688112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:40.787783 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:40.787756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:41.463100 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:41.463067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:41.553321 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:41.553295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:41.616301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:41.616274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:43.179458 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:43.179401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:48.964494 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:48.964460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:49.034858 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:49.034830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:49.584467 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:49.584425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:55.441130 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:55.441103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:55.539083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:55.539053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:55.670516 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:55.670493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:55.955094 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:55.955050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:56.149077 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:56.149046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:58.652105 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:58.652073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:58.754220 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:58.754193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:59.049603 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.049515 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62"] Jul 10 19:35:59.049987 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.049968 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd140e1f-18dc-4870-be14-7d74e9b9ca23" containerName="maas-api" Jul 10 19:35:59.050083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.049989 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd140e1f-18dc-4870-be14-7d74e9b9ca23" containerName="maas-api" Jul 10 19:35:59.050152 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.050137 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd140e1f-18dc-4870-be14-7d74e9b9ca23" containerName="maas-api" Jul 10 19:35:59.053472 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.053450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.056350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.056331 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8c3d0966-serving-cert\"" Jul 10 19:35:59.071666 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.071636 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62"] Jul 10 19:35:59.100801 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.100777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:59.136012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.135984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-openshift-service-ca\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.136117 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.136016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.136117 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.136053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhfw\" (UniqueName: \"kubernetes.io/projected/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-kube-api-access-pjhfw\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.236729 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.236684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-openshift-service-ca\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.236729 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.236736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.236993 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.236802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhfw\" (UniqueName: \"kubernetes.io/projected/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-kube-api-access-pjhfw\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.236993 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:35:59.236901 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-serving-cert: secret "maas-api-e2e-mt-8c3d0966-serving-cert" not found Jul 10 19:35:59.236993 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:35:59.236963 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls podName:73c8e7ce-7bb2-414d-a93f-3185445b0c2c nodeName:}" failed. No retries permitted until 2026-07-10 19:35:59.736944935 +0000 UTC m=+1699.775817929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls") pod "maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" (UID: "73c8e7ce-7bb2-414d-a93f-3185445b0c2c") : secret "maas-api-e2e-mt-8c3d0966-serving-cert" not found Jul 10 19:35:59.237344 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.237323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-openshift-service-ca\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.245970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.245955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhfw\" (UniqueName: \"kubernetes.io/projected/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-kube-api-access-pjhfw\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.334505 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.334480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:59.644530 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.644450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:59.741754 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.741727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.744016 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.743996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls\") pod \"maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:35:59.745211 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.745185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:59.964253 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:35:59.964191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:36:00.095161 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:00.095137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62"] Jul 10 19:36:00.097378 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:36:00.097341 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c8e7ce_7bb2_414d_a93f_3185445b0c2c.slice/crio-ebed8797ed063b71c6871323940f78f86bc1a88a6992f36cca3b2037179ed898 WatchSource:0}: Error finding container ebed8797ed063b71c6871323940f78f86bc1a88a6992f36cca3b2037179ed898: Status 404 returned error can't find the container with id ebed8797ed063b71c6871323940f78f86bc1a88a6992f36cca3b2037179ed898 Jul 10 19:36:00.462266 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:00.462237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" event={"ID":"73c8e7ce-7bb2-414d-a93f-3185445b0c2c","Type":"ContainerStarted","Data":"ebed8797ed063b71c6871323940f78f86bc1a88a6992f36cca3b2037179ed898"} Jul 10 19:36:01.468418 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:01.468382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" event={"ID":"73c8e7ce-7bb2-414d-a93f-3185445b0c2c","Type":"ContainerStarted","Data":"5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a"} Jul 10 19:36:01.468910 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:01.468504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:36:01.487116 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:01.487071 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" podStartSLOduration=2.0792350649999998 podStartE2EDuration="2.487058121s" totalImagesPullingTime="407.823056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:36:01.485408589 +0000 UTC m=+1701.524281627" watchObservedRunningTime="2026-07-10 19:36:01.487058121 +0000 UTC m=+1701.525931138" Jul 10 19:36:03.131117 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:03.131041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:03.787953 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:03.787920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:03.887942 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:03.887915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:04.212852 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:04.212824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:07.478721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:07.478695 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:36:09.768047 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:09.768013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:09.869716 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:09.869687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:10.176776 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.176745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:10.488522 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.488433 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62"] Jul 10 19:36:10.488731 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.488703 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" podUID="73c8e7ce-7bb2-414d-a93f-3185445b0c2c" containerName="maas-api" containerID="cri-o://5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a" gracePeriod=30 Jul 10 19:36:10.523553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.523530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:10.623839 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.623809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:10.732929 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.732904 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:36:10.740426 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.740383 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-openshift-service-ca\") pod \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " Jul 10 19:36:10.740426 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.740415 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjhfw\" (UniqueName: \"kubernetes.io/projected/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-kube-api-access-pjhfw\") pod \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " Jul 10 19:36:10.740539 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.740525 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls\") pod \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\" (UID: \"73c8e7ce-7bb2-414d-a93f-3185445b0c2c\") " Jul 10 19:36:10.740794 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.740765 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "73c8e7ce-7bb2-414d-a93f-3185445b0c2c" (UID: "73c8e7ce-7bb2-414d-a93f-3185445b0c2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:36:10.742352 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.742327 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-kube-api-access-pjhfw" (OuterVolumeSpecName: "kube-api-access-pjhfw") pod "73c8e7ce-7bb2-414d-a93f-3185445b0c2c" (UID: "73c8e7ce-7bb2-414d-a93f-3185445b0c2c"). InnerVolumeSpecName "kube-api-access-pjhfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:36:10.742502 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.742483 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "73c8e7ce-7bb2-414d-a93f-3185445b0c2c" (UID: "73c8e7ce-7bb2-414d-a93f-3185445b0c2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:36:10.841984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.841954 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:36:10.841984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.841980 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:36:10.842330 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:10.841994 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhfw\" (UniqueName: \"kubernetes.io/projected/73c8e7ce-7bb2-414d-a93f-3185445b0c2c-kube-api-access-pjhfw\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:36:11.515698 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.515668 2579 generic.go:356] "Generic (PLEG): container finished" podID="73c8e7ce-7bb2-414d-a93f-3185445b0c2c" containerID="5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a" exitCode=0 Jul 10 19:36:11.515843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.515714 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" Jul 10 19:36:11.515843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.515762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" event={"ID":"73c8e7ce-7bb2-414d-a93f-3185445b0c2c","Type":"ContainerDied","Data":"5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a"} Jul 10 19:36:11.515843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.515814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62" event={"ID":"73c8e7ce-7bb2-414d-a93f-3185445b0c2c","Type":"ContainerDied","Data":"ebed8797ed063b71c6871323940f78f86bc1a88a6992f36cca3b2037179ed898"} Jul 10 19:36:11.515843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.515842 2579 scope.go:117] "RemoveContainer" containerID="5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a" Jul 10 19:36:11.524451 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.524434 2579 scope.go:117] "RemoveContainer" containerID="5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a" Jul 10 19:36:11.524729 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:36:11.524710 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a\": container with ID starting with 5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a not found: ID does not exist" containerID="5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a" Jul 10 19:36:11.524784 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.524739 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a"} err="failed to get container status \"5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a\": rpc error: code = NotFound desc = could not find container \"5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a\": container with ID starting with 5faf7c2f2911f648106ed8d8e649d6d65744c0dc4c8225f98e2314a3b76c1b4a not found: ID does not exist" Jul 10 19:36:11.532939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.532917 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62"] Jul 10 19:36:11.538765 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:11.538746 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c3d0966-74db6f485c-kgj62"] Jul 10 19:36:13.076305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:13.076271 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c8e7ce-7bb2-414d-a93f-3185445b0c2c" path="/var/lib/kubelet/pods/73c8e7ce-7bb2-414d-a93f-3185445b0c2c/volumes" Jul 10 19:36:16.514036 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:16.513992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:16.613851 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:16.613804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:16.704210 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:16.704158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:16.912313 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:16.912288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:17.018198 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:17.018170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:17.126472 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:17.126442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:17.223644 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:17.223566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:22.747155 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:22.747119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:22.851513 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:22.851484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:23.004742 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:23.004645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:23.091053 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:23.091027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:28.628638 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:28.628573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:28.729218 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:28.729193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:35.421102 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:35.421051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:35.777321 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:35.777238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:36.040382 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:36.040304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:38.813972 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.813937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:38.913124 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.913097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:38.984164 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.984134 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889"] Jul 10 19:36:38.984722 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.984704 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c8e7ce-7bb2-414d-a93f-3185445b0c2c" containerName="maas-api" Jul 10 19:36:38.984775 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.984726 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c8e7ce-7bb2-414d-a93f-3185445b0c2c" containerName="maas-api" Jul 10 19:36:38.984870 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.984858 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c8e7ce-7bb2-414d-a93f-3185445b0c2c" containerName="maas-api" Jul 10 19:36:38.989793 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.989777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:38.992903 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:38.992884 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fbace407-serving-cert\"" Jul 10 19:36:39.001568 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.001546 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889"] Jul 10 19:36:39.023759 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.023735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:39.075063 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.075004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c94n2\" (UniqueName: \"kubernetes.io/projected/262f79d1-9520-4d4c-b32c-5d983b561961-kube-api-access-c94n2\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.075152 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.075062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f79d1-9520-4d4c-b32c-5d983b561961-openshift-service-ca\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.075197 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.075144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f79d1-9520-4d4c-b32c-5d983b561961-maas-api-tls\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.175707 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.175670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f79d1-9520-4d4c-b32c-5d983b561961-openshift-service-ca\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.175833 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.175732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f79d1-9520-4d4c-b32c-5d983b561961-maas-api-tls\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.175833 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.175755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94n2\" (UniqueName: \"kubernetes.io/projected/262f79d1-9520-4d4c-b32c-5d983b561961-kube-api-access-c94n2\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.176405 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.176384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f79d1-9520-4d4c-b32c-5d983b561961-openshift-service-ca\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.178075 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.178058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f79d1-9520-4d4c-b32c-5d983b561961-maas-api-tls\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.192470 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.192444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94n2\" (UniqueName: \"kubernetes.io/projected/262f79d1-9520-4d4c-b32c-5d983b561961-kube-api-access-c94n2\") pod \"maas-api-e2e-mt-fbace407-65f96b798b-2c889\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.194778 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.194759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:39.300261 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.300227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:39.430251 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.430229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889"] Jul 10 19:36:39.432619 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:36:39.432587 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262f79d1_9520_4d4c_b32c_5d983b561961.slice/crio-bae821e09c1daa22e6ff59587f8e8ae680ca148a6b2308c5f570976671b19560 WatchSource:0}: Error finding container bae821e09c1daa22e6ff59587f8e8ae680ca148a6b2308c5f570976671b19560: Status 404 returned error can't find the container with id bae821e09c1daa22e6ff59587f8e8ae680ca148a6b2308c5f570976671b19560 Jul 10 19:36:39.543640 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.543604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:39.644225 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.644137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:39.646940 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:39.646915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" event={"ID":"262f79d1-9520-4d4c-b32c-5d983b561961","Type":"ContainerStarted","Data":"bae821e09c1daa22e6ff59587f8e8ae680ca148a6b2308c5f570976671b19560"} Jul 10 19:36:40.652905 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:40.652865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" event={"ID":"262f79d1-9520-4d4c-b32c-5d983b561961","Type":"ContainerStarted","Data":"df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e"} Jul 10 19:36:40.653361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:40.653005 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:40.671096 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:40.671054 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" podStartSLOduration=2.05718357 podStartE2EDuration="2.671041026s" totalImagesPullingTime="613.857456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:36:40.668844465 +0000 UTC m=+1740.707717494" watchObservedRunningTime="2026-07-10 19:36:40.671041026 +0000 UTC m=+1740.709914042" Jul 10 19:36:43.342869 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:43.342838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:46.662245 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:46.662213 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:36:48.782251 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:48.782221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:48.836547 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:48.836518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:50.974481 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:50.974448 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4"] Jul 10 19:36:50.978315 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:50.978295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:50.981276 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:50.981254 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fbace407\"/\"openshift-service-ca.crt\"" Jul 10 19:36:50.982564 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:50.982543 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fbace407\"/\"kube-root-ca.crt\"" Jul 10 19:36:50.982722 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:50.982545 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fbace407\"/\"default-dockercfg-hfvcl\"" Jul 10 19:36:50.982722 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:50.982548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fbace407\"/\"e2e-lifecycle-model-fbace407-kserve-self-signed-certs\"" Jul 10 19:36:50.987469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:50.987445 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4"] Jul 10 19:36:51.004862 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.004834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:51.091239 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.091213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-dshm\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.091350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.091257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tmp-dir\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.091350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.091277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fvs\" (UniqueName: \"kubernetes.io/projected/aeca1507-df72-49aa-b3e4-d335c76e0b3c-kube-api-access-24fvs\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.091350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.091328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tls-certs\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.091350 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.091344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-home\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.091564 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.091461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-model-cache\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.192829 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.192805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tls-certs\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.192928 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.192833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-home\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.192928 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.192874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-model-cache\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.192928 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.192918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-dshm\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.193091 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.192942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tmp-dir\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.193091 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.192964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fvs\" (UniqueName: \"kubernetes.io/projected/aeca1507-df72-49aa-b3e4-d335c76e0b3c-kube-api-access-24fvs\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.193256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.193237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-home\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.193317 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.193277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tmp-dir\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.193355 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.193317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-model-cache\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.195099 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.195073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-dshm\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.195339 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.195322 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tls-certs\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.200759 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.200726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fvs\" (UniqueName: \"kubernetes.io/projected/aeca1507-df72-49aa-b3e4-d335c76e0b3c-kube-api-access-24fvs\") pod \"e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.289173 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.289115 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:36:51.421086 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.421065 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4"] Jul 10 19:36:51.423030 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:36:51.423001 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeca1507_df72_49aa_b3e4_d335c76e0b3c.slice/crio-d472ec33434a7d8f33d29f9e7005c44189ba666756da6a6ce1a3dc1c4f433a65 WatchSource:0}: Error finding container d472ec33434a7d8f33d29f9e7005c44189ba666756da6a6ce1a3dc1c4f433a65: Status 404 returned error can't find the container with id d472ec33434a7d8f33d29f9e7005c44189ba666756da6a6ce1a3dc1c4f433a65 Jul 10 19:36:51.709999 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.709966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" event={"ID":"aeca1507-df72-49aa-b3e4-d335c76e0b3c","Type":"ContainerStarted","Data":"b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194"} Jul 10 19:36:51.710156 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.710014 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" event={"ID":"aeca1507-df72-49aa-b3e4-d335c76e0b3c","Type":"ContainerStarted","Data":"d472ec33434a7d8f33d29f9e7005c44189ba666756da6a6ce1a3dc1c4f433a65"} Jul 10 19:36:51.730886 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:36:51.730841 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" podStartSLOduration=1.730825858 podStartE2EDuration="1.730825858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:36:51.730564108 +0000 UTC m=+1751.769437125" watchObservedRunningTime="2026-07-10 19:36:51.730825858 +0000 UTC m=+1751.769698875" Jul 10 19:37:01.290156 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:01.290127 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:37:01.290619 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:01.290169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:37:01.302424 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:01.302400 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:37:01.767851 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:01.767823 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:37:08.171981 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:08.171951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:08.271844 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:08.271813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:10.874892 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:10.874858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:11.447772 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:11.447728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:11.553761 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:11.553732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:11.698594 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:11.698518 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889"] Jul 10 19:37:11.698762 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:11.698729 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" podUID="262f79d1-9520-4d4c-b32c-5d983b561961" containerName="maas-api" containerID="cri-o://df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e" gracePeriod=30 Jul 10 19:37:11.725874 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:11.725851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:11.940906 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:11.940877 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:37:12.065514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.065444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f79d1-9520-4d4c-b32c-5d983b561961-maas-api-tls\") pod \"262f79d1-9520-4d4c-b32c-5d983b561961\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " Jul 10 19:37:12.065514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.065496 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c94n2\" (UniqueName: \"kubernetes.io/projected/262f79d1-9520-4d4c-b32c-5d983b561961-kube-api-access-c94n2\") pod \"262f79d1-9520-4d4c-b32c-5d983b561961\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " Jul 10 19:37:12.065759 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.065527 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f79d1-9520-4d4c-b32c-5d983b561961-openshift-service-ca\") pod \"262f79d1-9520-4d4c-b32c-5d983b561961\" (UID: \"262f79d1-9520-4d4c-b32c-5d983b561961\") " Jul 10 19:37:12.065960 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.065934 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262f79d1-9520-4d4c-b32c-5d983b561961-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "262f79d1-9520-4d4c-b32c-5d983b561961" (UID: "262f79d1-9520-4d4c-b32c-5d983b561961"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:37:12.067346 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.067326 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262f79d1-9520-4d4c-b32c-5d983b561961-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "262f79d1-9520-4d4c-b32c-5d983b561961" (UID: "262f79d1-9520-4d4c-b32c-5d983b561961"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:37:12.067528 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.067502 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262f79d1-9520-4d4c-b32c-5d983b561961-kube-api-access-c94n2" (OuterVolumeSpecName: "kube-api-access-c94n2") pod "262f79d1-9520-4d4c-b32c-5d983b561961" (UID: "262f79d1-9520-4d4c-b32c-5d983b561961"). InnerVolumeSpecName "kube-api-access-c94n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:37:12.166697 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.166678 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262f79d1-9520-4d4c-b32c-5d983b561961-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.166787 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.166698 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c94n2\" (UniqueName: \"kubernetes.io/projected/262f79d1-9520-4d4c-b32c-5d983b561961-kube-api-access-c94n2\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.166787 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.166717 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262f79d1-9520-4d4c-b32c-5d983b561961-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.808899 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.808870 2579 generic.go:356] "Generic (PLEG): container finished" podID="262f79d1-9520-4d4c-b32c-5d983b561961" containerID="df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e" exitCode=0 Jul 10 19:37:12.809074 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.808914 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" Jul 10 19:37:12.809074 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.808960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" event={"ID":"262f79d1-9520-4d4c-b32c-5d983b561961","Type":"ContainerDied","Data":"df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e"} Jul 10 19:37:12.809074 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.809010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889" event={"ID":"262f79d1-9520-4d4c-b32c-5d983b561961","Type":"ContainerDied","Data":"bae821e09c1daa22e6ff59587f8e8ae680ca148a6b2308c5f570976671b19560"} Jul 10 19:37:12.809074 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.809032 2579 scope.go:117] "RemoveContainer" containerID="df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e" Jul 10 19:37:12.818329 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.818313 2579 scope.go:117] "RemoveContainer" containerID="df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e" Jul 10 19:37:12.818590 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:37:12.818573 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e\": container with ID starting with df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e not found: ID does not exist" containerID="df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e" Jul 10 19:37:12.818633 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.818598 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e"} err="failed to get container status \"df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e\": rpc error: code = NotFound desc = could not find container \"df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e\": container with ID starting with df6646946670042ad8734a6bccd3a5bdf1ec3d6c278fbfa5f848059610a2655e not found: ID does not exist" Jul 10 19:37:12.836843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.834947 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889"] Jul 10 19:37:12.839032 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:12.839010 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fbace407-65f96b798b-2c889"] Jul 10 19:37:13.070512 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:13.070422 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="262f79d1-9520-4d4c-b32c-5d983b561961" path="/var/lib/kubelet/pods/262f79d1-9520-4d4c-b32c-5d983b561961/volumes" Jul 10 19:37:17.028199 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:17.028134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:22.008513 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:22.008483 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4"] Jul 10 19:37:22.103647 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:22.103616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:37.028147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.028049 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" podUID="aeca1507-df72-49aa-b3e4-d335c76e0b3c" containerName="main" containerID="cri-o://b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194" gracePeriod=45 Jul 10 19:37:37.275819 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.275796 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:37:37.389242 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389202 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tls-certs\") pod \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " Jul 10 19:37:37.389242 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389248 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-home\") pod \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " Jul 10 19:37:37.389465 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-model-cache\") pod \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " Jul 10 19:37:37.389529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389470 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-home" (OuterVolumeSpecName: "home") pod "aeca1507-df72-49aa-b3e4-d335c76e0b3c" (UID: "aeca1507-df72-49aa-b3e4-d335c76e0b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:37.389529 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389492 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tmp-dir\") pod \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " Jul 10 19:37:37.389634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-dshm\") pod \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " Jul 10 19:37:37.389634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389563 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24fvs\" (UniqueName: \"kubernetes.io/projected/aeca1507-df72-49aa-b3e4-d335c76e0b3c-kube-api-access-24fvs\") pod \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\" (UID: \"aeca1507-df72-49aa-b3e4-d335c76e0b3c\") " Jul 10 19:37:37.389634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeca1507-df72-49aa-b3e4-d335c76e0b3c" (UID: "aeca1507-df72-49aa-b3e4-d335c76e0b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:37.389799 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389767 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeca1507-df72-49aa-b3e4-d335c76e0b3c" (UID: "aeca1507-df72-49aa-b3e4-d335c76e0b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:37.389979 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389963 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:37.390045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389981 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:37.390045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.389991 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:37.391454 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.391430 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeca1507-df72-49aa-b3e4-d335c76e0b3c" (UID: "aeca1507-df72-49aa-b3e4-d335c76e0b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:37:37.391591 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.391546 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-dshm" (OuterVolumeSpecName: "dshm") pod "aeca1507-df72-49aa-b3e4-d335c76e0b3c" (UID: "aeca1507-df72-49aa-b3e4-d335c76e0b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:37.391591 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.391568 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeca1507-df72-49aa-b3e4-d335c76e0b3c-kube-api-access-24fvs" (OuterVolumeSpecName: "kube-api-access-24fvs") pod "aeca1507-df72-49aa-b3e4-d335c76e0b3c" (UID: "aeca1507-df72-49aa-b3e4-d335c76e0b3c"). InnerVolumeSpecName "kube-api-access-24fvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:37:37.490427 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.490406 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeca1507-df72-49aa-b3e4-d335c76e0b3c-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:37.490427 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.490428 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fvs\" (UniqueName: \"kubernetes.io/projected/aeca1507-df72-49aa-b3e4-d335c76e0b3c-kube-api-access-24fvs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:37.490572 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.490439 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeca1507-df72-49aa-b3e4-d335c76e0b3c-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:37:37.926197 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.926170 2579 generic.go:356] "Generic (PLEG): container finished" podID="aeca1507-df72-49aa-b3e4-d335c76e0b3c" containerID="b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194" exitCode=0 Jul 10 19:37:37.926364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.926227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" event={"ID":"aeca1507-df72-49aa-b3e4-d335c76e0b3c","Type":"ContainerDied","Data":"b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194"} Jul 10 19:37:37.926364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.926255 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" event={"ID":"aeca1507-df72-49aa-b3e4-d335c76e0b3c","Type":"ContainerDied","Data":"d472ec33434a7d8f33d29f9e7005c44189ba666756da6a6ce1a3dc1c4f433a65"} Jul 10 19:37:37.926364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.926225 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4" Jul 10 19:37:37.926364 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.926276 2579 scope.go:117] "RemoveContainer" containerID="b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194" Jul 10 19:37:37.934847 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.934828 2579 scope.go:117] "RemoveContainer" containerID="b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194" Jul 10 19:37:37.935070 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:37:37.935050 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194\": container with ID starting with b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194 not found: ID does not exist" containerID="b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194" Jul 10 19:37:37.935126 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.935076 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194"} err="failed to get container status \"b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194\": rpc error: code = NotFound desc = could not find container \"b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194\": container with ID starting with b1ec3f46f57d8b02d21d2d9c99b175c1b660fa4e29f04bad786272ad3925e194 not found: ID does not exist" Jul 10 19:37:37.950837 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.950815 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4"] Jul 10 19:37:37.956428 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:37.956409 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fbace407/e2e-lifecycle-model-fbace407-kserve-859f9d8c9f-b66c4"] Jul 10 19:37:39.070936 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:39.070903 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeca1507-df72-49aa-b3e4-d335c76e0b3c" path="/var/lib/kubelet/pods/aeca1507-df72-49aa-b3e4-d335c76e0b3c/volumes" Jul 10 19:37:41.012011 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:41.011988 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:37:41.017404 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:41.017385 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:37:58.806163 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:58.806131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:59.217620 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:37:59.217584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:04.989692 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:04.989638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:19.870044 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:19.870010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:20.972969 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:20.972934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:21.694905 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:21.694852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:21.790023 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:21.789993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:21.888808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:21.888779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:22.790579 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:22.790548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:22.890134 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:22.890105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:23.002476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:23.002449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:24.705258 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:24.705218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:30.802198 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:30.802162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:30.899749 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:30.899720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:31.612506 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:31.612475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:38.248268 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:38.248238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:38.365845 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:38.365818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:38.468099 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:38.468069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:49.114592 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:49.114556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:49.506759 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:49.506675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:54.855998 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:38:54.855968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:08.072353 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:08.072315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:08.128956 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:08.128929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:10.460128 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:10.460091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:16.329253 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:16.329224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:16.388473 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:16.388446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:16.681903 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:16.681876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:16.992551 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:16.992464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:17.093071 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:17.093034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:20.405636 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.405603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:20.495388 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.495356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:20.499826 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.499800 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl"] Jul 10 19:39:20.500212 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.500195 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="262f79d1-9520-4d4c-b32c-5d983b561961" containerName="maas-api" Jul 10 19:39:20.500289 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.500214 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="262f79d1-9520-4d4c-b32c-5d983b561961" containerName="maas-api" Jul 10 19:39:20.500289 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.500265 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeca1507-df72-49aa-b3e4-d335c76e0b3c" containerName="main" Jul 10 19:39:20.500289 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.500274 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeca1507-df72-49aa-b3e4-d335c76e0b3c" containerName="main" Jul 10 19:39:20.500440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.500360 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="262f79d1-9520-4d4c-b32c-5d983b561961" containerName="maas-api" Jul 10 19:39:20.500440 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.500378 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeca1507-df72-49aa-b3e4-d335c76e0b3c" containerName="main" Jul 10 19:39:20.503450 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.503433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.506262 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.506247 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dac879-serving-cert\"" Jul 10 19:39:20.514147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.514124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl"] Jul 10 19:39:20.594796 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.594768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:20.653638 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.653616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6z9\" (UniqueName: \"kubernetes.io/projected/2180e11d-b566-44e0-89a7-c2695bc70b3c-kube-api-access-5n6z9\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.653747 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.653643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2180e11d-b566-44e0-89a7-c2695bc70b3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.653747 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.653734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2180e11d-b566-44e0-89a7-c2695bc70b3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.718489 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.718430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:20.754689 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.754647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2180e11d-b566-44e0-89a7-c2695bc70b3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.754786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.754744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6z9\" (UniqueName: \"kubernetes.io/projected/2180e11d-b566-44e0-89a7-c2695bc70b3c-kube-api-access-5n6z9\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.754786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.754761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2180e11d-b566-44e0-89a7-c2695bc70b3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.755293 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.755273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2180e11d-b566-44e0-89a7-c2695bc70b3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.757127 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.757106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2180e11d-b566-44e0-89a7-c2695bc70b3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.763821 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.763803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6z9\" (UniqueName: \"kubernetes.io/projected/2180e11d-b566-44e0-89a7-c2695bc70b3c-kube-api-access-5n6z9\") pod \"maas-api-e2e-inf-a-dac879-85664f7494-fwwsl\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.813810 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.813790 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:20.949171 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.949137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl"] Jul 10 19:39:20.951584 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:39:20.951546 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2180e11d_b566_44e0_89a7_c2695bc70b3c.slice/crio-3d2a091c7f0db7834ec0236d72e208a4aec4bfa8939f18b64e2ed641da4685dc WatchSource:0}: Error finding container 3d2a091c7f0db7834ec0236d72e208a4aec4bfa8939f18b64e2ed641da4685dc: Status 404 returned error can't find the container with id 3d2a091c7f0db7834ec0236d72e208a4aec4bfa8939f18b64e2ed641da4685dc Jul 10 19:39:20.952970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:20.952951 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:39:21.176280 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:21.176247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:21.277021 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:21.276991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:21.398771 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:21.398741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" event={"ID":"2180e11d-b566-44e0-89a7-c2695bc70b3c","Type":"ContainerStarted","Data":"3d2a091c7f0db7834ec0236d72e208a4aec4bfa8939f18b64e2ed641da4685dc"} Jul 10 19:39:22.404750 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:22.404718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" event={"ID":"2180e11d-b566-44e0-89a7-c2695bc70b3c","Type":"ContainerStarted","Data":"b33d1a85b0aa818c6d33fef639f639049d0aeb84d5c88bfba685821d6cfb6de9"} Jul 10 19:39:22.405117 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:22.404773 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:22.421301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:22.421258 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" podStartSLOduration=1.900030352 podStartE2EDuration="2.421243123s" totalImagesPullingTime="521.212771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:39:22.420240869 +0000 UTC m=+1902.459113885" watchObservedRunningTime="2026-07-10 19:39:22.421243123 +0000 UTC m=+1902.460116141" Jul 10 19:39:24.900346 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:24.900310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:25.238456 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:25.238383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:25.563842 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:25.563763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:25.677769 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:25.677738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:28.396639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.396607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:28.414064 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.414045 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:39:28.482716 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.482687 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv"] Jul 10 19:39:28.486134 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.486102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:28.486497 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.486481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.489765 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.489745 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-43c766-serving-cert\"" Jul 10 19:39:28.495707 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.495683 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv"] Jul 10 19:39:28.514820 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.514795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2cg\" (UniqueName: \"kubernetes.io/projected/d715453e-fb6a-49db-86bd-35e99c1befa1-kube-api-access-hv2cg\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.514936 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.514834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.514936 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.514896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d715453e-fb6a-49db-86bd-35e99c1befa1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.580989 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.580963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:28.616205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.616185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2cg\" (UniqueName: \"kubernetes.io/projected/d715453e-fb6a-49db-86bd-35e99c1befa1-kube-api-access-hv2cg\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.616299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.616219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.616299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.616263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d715453e-fb6a-49db-86bd-35e99c1befa1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.616400 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:39:28.616384 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-serving-cert: secret "maas-api-e2e-inf-b-43c766-serving-cert" not found Jul 10 19:39:28.616447 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:39:28.616438 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls podName:d715453e-fb6a-49db-86bd-35e99c1befa1 nodeName:}" failed. No retries permitted until 2026-07-10 19:39:29.116423771 +0000 UTC m=+1909.155296766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls") pod "maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" (UID: "d715453e-fb6a-49db-86bd-35e99c1befa1") : secret "maas-api-e2e-inf-b-43c766-serving-cert" not found Jul 10 19:39:28.616887 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.616870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d715453e-fb6a-49db-86bd-35e99c1befa1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:28.634692 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:28.634667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2cg\" (UniqueName: \"kubernetes.io/projected/d715453e-fb6a-49db-86bd-35e99c1befa1-kube-api-access-hv2cg\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:29.072733 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:29.072707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:29.120275 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:29.120243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:29.122813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:29.122792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls\") pod \"maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:29.173992 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:29.173962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:29.294897 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:29.294875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:29.399898 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:29.399873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:29.534186 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:29.534156 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv"] Jul 10 19:39:30.448948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:30.448917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" event={"ID":"d715453e-fb6a-49db-86bd-35e99c1befa1","Type":"ContainerStarted","Data":"319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587"} Jul 10 19:39:30.449301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:30.448966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" event={"ID":"d715453e-fb6a-49db-86bd-35e99c1befa1","Type":"ContainerStarted","Data":"8cdfe6e43fb6220d4f5b9fcabcc6138ea366125b2c33eeb136b472c8166283e9"} Jul 10 19:39:30.449301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:30.449004 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:30.468475 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:30.468426 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" podStartSLOduration=2.076628352 podStartE2EDuration="2.468411631s" totalImagesPullingTime="391.783279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:39:30.465433491 +0000 UTC m=+1910.504306509" watchObservedRunningTime="2026-07-10 19:39:30.468411631 +0000 UTC m=+1910.507284648" Jul 10 19:39:32.876310 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:32.876276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:36.458965 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:36.458937 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:39:39.874466 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.874433 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf"] Jul 10 19:39:39.878281 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.878259 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:39.881101 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.881080 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dac879\"/\"openshift-service-ca.crt\"" Jul 10 19:39:39.882564 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.882547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dac879\"/\"test-model-dac879-kserve-self-signed-certs\"" Jul 10 19:39:39.882675 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.882613 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dac879\"/\"kube-root-ca.crt\"" Jul 10 19:39:39.882753 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.882730 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dac879\"/\"default-dockercfg-sd5kv\"" Jul 10 19:39:39.886360 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.886338 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf"] Jul 10 19:39:39.898963 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.898941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:39.918351 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.918330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-model-cache\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:39.918444 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.918366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-dshm\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:39.918490 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.918457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/b84cd1af-f705-4e66-8487-74b79c334f6c-kube-api-access-qbhlj\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:39.918526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.918508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-home\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:39.918560 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.918527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84cd1af-f705-4e66-8487-74b79c334f6c-tls-certs\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:39.918607 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:39.918568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-tmp-dir\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.019482 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.019459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-dshm\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.019591 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.019513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/b84cd1af-f705-4e66-8487-74b79c334f6c-kube-api-access-qbhlj\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.019591 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.019583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-home\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.019742 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.019611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84cd1af-f705-4e66-8487-74b79c334f6c-tls-certs\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.019742 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.019685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-tmp-dir\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.019857 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.019743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-model-cache\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.019992 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.019961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-home\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.020090 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.020022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-model-cache\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.020090 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.020060 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-tmp-dir\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.021716 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.021697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-dshm\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.021950 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.021933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84cd1af-f705-4e66-8487-74b79c334f6c-tls-certs\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.027685 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.027648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/b84cd1af-f705-4e66-8487-74b79c334f6c-kube-api-access-qbhlj\") pod \"test-model-dac879-kserve-dfc85fd45-9qjjf\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.188101 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.188036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:40.318461 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.318437 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf"] Jul 10 19:39:40.320425 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:39:40.320398 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb84cd1af_f705_4e66_8487_74b79c334f6c.slice/crio-2c4a55d75ff15c5e26423207774bc425c845b500f5983efee92032c4014fbfe7 WatchSource:0}: Error finding container 2c4a55d75ff15c5e26423207774bc425c845b500f5983efee92032c4014fbfe7: Status 404 returned error can't find the container with id 2c4a55d75ff15c5e26423207774bc425c845b500f5983efee92032c4014fbfe7 Jul 10 19:39:40.498360 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.498287 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" event={"ID":"b84cd1af-f705-4e66-8487-74b79c334f6c","Type":"ContainerStarted","Data":"6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3"} Jul 10 19:39:40.498360 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.498323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" event={"ID":"b84cd1af-f705-4e66-8487-74b79c334f6c","Type":"ContainerStarted","Data":"2c4a55d75ff15c5e26423207774bc425c845b500f5983efee92032c4014fbfe7"} Jul 10 19:39:40.516918 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:40.516873 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" podStartSLOduration=1.516858149 podStartE2EDuration="1.516858149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:39:40.514708026 +0000 UTC m=+1920.553581033" watchObservedRunningTime="2026-07-10 19:39:40.516858149 +0000 UTC m=+1920.555731166" Jul 10 19:39:42.720952 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:42.720921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:42.821326 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:42.821295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:50.188618 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:50.188585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:50.190949 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:50.188630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:50.200841 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:50.200823 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:50.557521 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:50.557447 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:39:50.861725 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:50.861693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:50.962699 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:39:50.962670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:21.374296 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.374266 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l"] Jul 10 19:40:21.378029 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.378007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.382233 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.382207 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-43c766\"/\"kube-root-ca.crt\"" Jul 10 19:40:21.382233 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.382230 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-43c766\"/\"openshift-service-ca.crt\"" Jul 10 19:40:21.383566 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.383544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-43c766\"/\"default-dockercfg-pk8d4\"" Jul 10 19:40:21.383724 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.383672 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-43c766\"/\"test-model-43c766-kserve-self-signed-certs\"" Jul 10 19:40:21.388961 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.388942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l"] Jul 10 19:40:21.399279 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.399244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:21.476832 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.476810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-dshm\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.476938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.476843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtwj\" (UniqueName: \"kubernetes.io/projected/07cf35ad-8c1c-4291-989c-fefeecf83efe-kube-api-access-whtwj\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.476938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.476862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-model-cache\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.476938 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.476894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cf35ad-8c1c-4291-989c-fefeecf83efe-tls-certs\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.477095 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.476939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-home\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.477095 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.476960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-tmp-dir\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.577676 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.577638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-home\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.577766 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.577680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-tmp-dir\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.577806 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.577763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-dshm\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.577806 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.577796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtwj\" (UniqueName: \"kubernetes.io/projected/07cf35ad-8c1c-4291-989c-fefeecf83efe-kube-api-access-whtwj\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.577908 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.577822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-model-cache\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.577908 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.577864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cf35ad-8c1c-4291-989c-fefeecf83efe-tls-certs\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.578087 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.578067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-home\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.578154 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.578121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-tmp-dir\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.578209 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.578170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-model-cache\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.580025 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.580001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-dshm\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.580215 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.580197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cf35ad-8c1c-4291-989c-fefeecf83efe-tls-certs\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.585746 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.585725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtwj\" (UniqueName: \"kubernetes.io/projected/07cf35ad-8c1c-4291-989c-fefeecf83efe-kube-api-access-whtwj\") pod \"test-model-43c766-kserve-7f67667cbc-9465l\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.688889 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.688833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:21.822639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:21.822598 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l"] Jul 10 19:40:21.823804 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:40:21.823777 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07cf35ad_8c1c_4291_989c_fefeecf83efe.slice/crio-8500897a57e010dd123e4026ec6b2b1e8c94ec339e9d558affa9af7c5a0ee45a WatchSource:0}: Error finding container 8500897a57e010dd123e4026ec6b2b1e8c94ec339e9d558affa9af7c5a0ee45a: Status 404 returned error can't find the container with id 8500897a57e010dd123e4026ec6b2b1e8c94ec339e9d558affa9af7c5a0ee45a Jul 10 19:40:22.709476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:22.709442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" event={"ID":"07cf35ad-8c1c-4291-989c-fefeecf83efe","Type":"ContainerStarted","Data":"35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841"} Jul 10 19:40:22.709476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:22.709485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" event={"ID":"07cf35ad-8c1c-4291-989c-fefeecf83efe","Type":"ContainerStarted","Data":"8500897a57e010dd123e4026ec6b2b1e8c94ec339e9d558affa9af7c5a0ee45a"} Jul 10 19:40:22.728968 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:22.728924 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" podStartSLOduration=1.728909123 podStartE2EDuration="1.728909123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:40:22.728560796 +0000 UTC m=+1962.767433813" watchObservedRunningTime="2026-07-10 19:40:22.728909123 +0000 UTC m=+1962.767782139" Jul 10 19:40:31.688955 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:31.688902 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:31.689424 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:31.688970 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:31.700823 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:31.700799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:31.762710 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:31.762684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:41.557384 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:41.557353 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf"] Jul 10 19:40:41.583843 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:41.583813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:42.559468 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:42.559440 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l"] Jul 10 19:40:42.582406 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:42.582378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:42.738931 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:42.738898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:43.135429 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.135395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:43.235561 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.235531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:43.334971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.334944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:43.424742 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.424678 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl"] Jul 10 19:40:43.424877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.424853 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" podUID="2180e11d-b566-44e0-89a7-c2695bc70b3c" containerName="maas-api" containerID="cri-o://b33d1a85b0aa818c6d33fef639f639049d0aeb84d5c88bfba685821d6cfb6de9" gracePeriod=30 Jul 10 19:40:43.450830 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.450807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:43.812794 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.812768 2579 generic.go:356] "Generic (PLEG): container finished" podID="2180e11d-b566-44e0-89a7-c2695bc70b3c" containerID="b33d1a85b0aa818c6d33fef639f639049d0aeb84d5c88bfba685821d6cfb6de9" exitCode=0 Jul 10 19:40:43.813127 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.812843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" event={"ID":"2180e11d-b566-44e0-89a7-c2695bc70b3c","Type":"ContainerDied","Data":"b33d1a85b0aa818c6d33fef639f639049d0aeb84d5c88bfba685821d6cfb6de9"} Jul 10 19:40:43.873572 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.873550 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:40:43.969308 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.969221 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2180e11d-b566-44e0-89a7-c2695bc70b3c-maas-api-tls\") pod \"2180e11d-b566-44e0-89a7-c2695bc70b3c\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " Jul 10 19:40:43.969308 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.969285 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n6z9\" (UniqueName: \"kubernetes.io/projected/2180e11d-b566-44e0-89a7-c2695bc70b3c-kube-api-access-5n6z9\") pod \"2180e11d-b566-44e0-89a7-c2695bc70b3c\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " Jul 10 19:40:43.969519 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.969369 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2180e11d-b566-44e0-89a7-c2695bc70b3c-openshift-service-ca\") pod \"2180e11d-b566-44e0-89a7-c2695bc70b3c\" (UID: \"2180e11d-b566-44e0-89a7-c2695bc70b3c\") " Jul 10 19:40:43.969763 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.969736 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2180e11d-b566-44e0-89a7-c2695bc70b3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2180e11d-b566-44e0-89a7-c2695bc70b3c" (UID: "2180e11d-b566-44e0-89a7-c2695bc70b3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:40:43.969892 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.969877 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2180e11d-b566-44e0-89a7-c2695bc70b3c-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:43.971216 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.971193 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2180e11d-b566-44e0-89a7-c2695bc70b3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2180e11d-b566-44e0-89a7-c2695bc70b3c" (UID: "2180e11d-b566-44e0-89a7-c2695bc70b3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:40:43.971315 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:43.971300 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2180e11d-b566-44e0-89a7-c2695bc70b3c-kube-api-access-5n6z9" (OuterVolumeSpecName: "kube-api-access-5n6z9") pod "2180e11d-b566-44e0-89a7-c2695bc70b3c" (UID: "2180e11d-b566-44e0-89a7-c2695bc70b3c"). InnerVolumeSpecName "kube-api-access-5n6z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:44.070724 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:44.070692 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2180e11d-b566-44e0-89a7-c2695bc70b3c-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:44.070724 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:44.070720 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6z9\" (UniqueName: \"kubernetes.io/projected/2180e11d-b566-44e0-89a7-c2695bc70b3c-kube-api-access-5n6z9\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:44.819069 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:44.819043 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" Jul 10 19:40:44.819420 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:44.819042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl" event={"ID":"2180e11d-b566-44e0-89a7-c2695bc70b3c","Type":"ContainerDied","Data":"3d2a091c7f0db7834ec0236d72e208a4aec4bfa8939f18b64e2ed641da4685dc"} Jul 10 19:40:44.819420 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:44.819184 2579 scope.go:117] "RemoveContainer" containerID="b33d1a85b0aa818c6d33fef639f639049d0aeb84d5c88bfba685821d6cfb6de9" Jul 10 19:40:44.844887 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:44.844865 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl"] Jul 10 19:40:44.848957 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:44.848935 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dac879-85664f7494-fwwsl"] Jul 10 19:40:45.071062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:45.070987 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2180e11d-b566-44e0-89a7-c2695bc70b3c" path="/var/lib/kubelet/pods/2180e11d-b566-44e0-89a7-c2695bc70b3c/volumes" Jul 10 19:40:48.197939 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:48.197892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:56.577964 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.577920 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" podUID="b84cd1af-f705-4e66-8487-74b79c334f6c" containerName="main" containerID="cri-o://6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3" gracePeriod=45 Jul 10 19:40:56.828917 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.828862 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:40:56.878015 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.877980 2579 generic.go:356] "Generic (PLEG): container finished" podID="b84cd1af-f705-4e66-8487-74b79c334f6c" containerID="6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3" exitCode=0 Jul 10 19:40:56.878152 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.878023 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" Jul 10 19:40:56.878152 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.878066 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" event={"ID":"b84cd1af-f705-4e66-8487-74b79c334f6c","Type":"ContainerDied","Data":"6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3"} Jul 10 19:40:56.878152 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.878106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf" event={"ID":"b84cd1af-f705-4e66-8487-74b79c334f6c","Type":"ContainerDied","Data":"2c4a55d75ff15c5e26423207774bc425c845b500f5983efee92032c4014fbfe7"} Jul 10 19:40:56.878152 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.878124 2579 scope.go:117] "RemoveContainer" containerID="6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3" Jul 10 19:40:56.886354 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.886337 2579 scope.go:117] "RemoveContainer" containerID="6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3" Jul 10 19:40:56.886596 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:40:56.886576 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3\": container with ID starting with 6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3 not found: ID does not exist" containerID="6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3" Jul 10 19:40:56.886674 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.886603 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3"} err="failed to get container status \"6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3\": rpc error: code = NotFound desc = could not find container \"6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3\": container with ID starting with 6238b817801a0120ebc13f7b082a5a749f91f1c18ef88e894a1d85c7df42a2d3 not found: ID does not exist" Jul 10 19:40:56.978746 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.978726 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84cd1af-f705-4e66-8487-74b79c334f6c-tls-certs\") pod \"b84cd1af-f705-4e66-8487-74b79c334f6c\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " Jul 10 19:40:56.978863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.978786 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-model-cache\") pod \"b84cd1af-f705-4e66-8487-74b79c334f6c\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " Jul 10 19:40:56.978863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.978805 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-home\") pod \"b84cd1af-f705-4e66-8487-74b79c334f6c\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " Jul 10 19:40:56.978863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.978855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/b84cd1af-f705-4e66-8487-74b79c334f6c-kube-api-access-qbhlj\") pod \"b84cd1af-f705-4e66-8487-74b79c334f6c\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " Jul 10 19:40:56.979030 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.978951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-tmp-dir\") pod \"b84cd1af-f705-4e66-8487-74b79c334f6c\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " Jul 10 19:40:56.979030 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.978982 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-dshm\") pod \"b84cd1af-f705-4e66-8487-74b79c334f6c\" (UID: \"b84cd1af-f705-4e66-8487-74b79c334f6c\") " Jul 10 19:40:56.979127 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.979042 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b84cd1af-f705-4e66-8487-74b79c334f6c" (UID: "b84cd1af-f705-4e66-8487-74b79c334f6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:56.979127 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.979076 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-home" (OuterVolumeSpecName: "home") pod "b84cd1af-f705-4e66-8487-74b79c334f6c" (UID: "b84cd1af-f705-4e66-8487-74b79c334f6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:56.979251 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.979226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b84cd1af-f705-4e66-8487-74b79c334f6c" (UID: "b84cd1af-f705-4e66-8487-74b79c334f6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:56.979341 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.979322 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:56.979397 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.979345 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:56.979397 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.979357 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:56.981004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.980980 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b84cd1af-f705-4e66-8487-74b79c334f6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b84cd1af-f705-4e66-8487-74b79c334f6c" (UID: "b84cd1af-f705-4e66-8487-74b79c334f6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:40:56.981120 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.981029 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-dshm" (OuterVolumeSpecName: "dshm") pod "b84cd1af-f705-4e66-8487-74b79c334f6c" (UID: "b84cd1af-f705-4e66-8487-74b79c334f6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:56.981120 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:56.981036 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84cd1af-f705-4e66-8487-74b79c334f6c-kube-api-access-qbhlj" (OuterVolumeSpecName: "kube-api-access-qbhlj") pod "b84cd1af-f705-4e66-8487-74b79c334f6c" (UID: "b84cd1af-f705-4e66-8487-74b79c334f6c"). InnerVolumeSpecName "kube-api-access-qbhlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:57.080577 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.080520 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/b84cd1af-f705-4e66-8487-74b79c334f6c-kube-api-access-qbhlj\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.080577 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.080544 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84cd1af-f705-4e66-8487-74b79c334f6c-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.080577 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.080554 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84cd1af-f705-4e66-8487-74b79c334f6c-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.193786 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.193757 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf"] Jul 10 19:40:57.197191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.197168 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dac879/test-model-dac879-kserve-dfc85fd45-9qjjf"] Jul 10 19:40:57.579840 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.579806 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" podUID="07cf35ad-8c1c-4291-989c-fefeecf83efe" containerName="main" containerID="cri-o://35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841" gracePeriod=45 Jul 10 19:40:57.825306 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.825282 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:57.884114 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.884087 2579 generic.go:356] "Generic (PLEG): container finished" podID="07cf35ad-8c1c-4291-989c-fefeecf83efe" containerID="35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841" exitCode=0 Jul 10 19:40:57.884266 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.884128 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" Jul 10 19:40:57.884266 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.884167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" event={"ID":"07cf35ad-8c1c-4291-989c-fefeecf83efe","Type":"ContainerDied","Data":"35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841"} Jul 10 19:40:57.884266 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.884211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l" event={"ID":"07cf35ad-8c1c-4291-989c-fefeecf83efe","Type":"ContainerDied","Data":"8500897a57e010dd123e4026ec6b2b1e8c94ec339e9d558affa9af7c5a0ee45a"} Jul 10 19:40:57.884266 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.884235 2579 scope.go:117] "RemoveContainer" containerID="35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841" Jul 10 19:40:57.889089 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889067 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whtwj\" (UniqueName: \"kubernetes.io/projected/07cf35ad-8c1c-4291-989c-fefeecf83efe-kube-api-access-whtwj\") pod \"07cf35ad-8c1c-4291-989c-fefeecf83efe\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " Jul 10 19:40:57.889256 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-tmp-dir\") pod \"07cf35ad-8c1c-4291-989c-fefeecf83efe\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " Jul 10 19:40:57.889317 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889276 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-home\") pod \"07cf35ad-8c1c-4291-989c-fefeecf83efe\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " Jul 10 19:40:57.889317 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889311 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cf35ad-8c1c-4291-989c-fefeecf83efe-tls-certs\") pod \"07cf35ad-8c1c-4291-989c-fefeecf83efe\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " Jul 10 19:40:57.889419 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889336 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-dshm\") pod \"07cf35ad-8c1c-4291-989c-fefeecf83efe\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " Jul 10 19:40:57.889419 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889378 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-model-cache\") pod \"07cf35ad-8c1c-4291-989c-fefeecf83efe\" (UID: \"07cf35ad-8c1c-4291-989c-fefeecf83efe\") " Jul 10 19:40:57.889574 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889552 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07cf35ad-8c1c-4291-989c-fefeecf83efe" (UID: "07cf35ad-8c1c-4291-989c-fefeecf83efe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:57.889637 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889571 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-home" (OuterVolumeSpecName: "home") pod "07cf35ad-8c1c-4291-989c-fefeecf83efe" (UID: "07cf35ad-8c1c-4291-989c-fefeecf83efe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:57.889750 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889719 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-model-cache" (OuterVolumeSpecName: "model-cache") pod "07cf35ad-8c1c-4291-989c-fefeecf83efe" (UID: "07cf35ad-8c1c-4291-989c-fefeecf83efe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:57.889885 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889827 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.889885 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889847 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.889885 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.889860 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.891434 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.891409 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-dshm" (OuterVolumeSpecName: "dshm") pod "07cf35ad-8c1c-4291-989c-fefeecf83efe" (UID: "07cf35ad-8c1c-4291-989c-fefeecf83efe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:57.891514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.891489 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cf35ad-8c1c-4291-989c-fefeecf83efe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07cf35ad-8c1c-4291-989c-fefeecf83efe" (UID: "07cf35ad-8c1c-4291-989c-fefeecf83efe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:40:57.891514 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.891498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cf35ad-8c1c-4291-989c-fefeecf83efe-kube-api-access-whtwj" (OuterVolumeSpecName: "kube-api-access-whtwj") pod "07cf35ad-8c1c-4291-989c-fefeecf83efe" (UID: "07cf35ad-8c1c-4291-989c-fefeecf83efe"). InnerVolumeSpecName "kube-api-access-whtwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:57.892808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.892792 2579 scope.go:117] "RemoveContainer" containerID="35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841" Jul 10 19:40:57.893032 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:40:57.893015 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841\": container with ID starting with 35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841 not found: ID does not exist" containerID="35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841" Jul 10 19:40:57.893073 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.893040 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841"} err="failed to get container status \"35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841\": rpc error: code = NotFound desc = could not find container \"35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841\": container with ID starting with 35ad6cf16fcca181baa9913f390279d1cd610cef834b75ca6160cb0b50e37841 not found: ID does not exist" Jul 10 19:40:57.991033 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.991006 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07cf35ad-8c1c-4291-989c-fefeecf83efe-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.991033 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.991030 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07cf35ad-8c1c-4291-989c-fefeecf83efe-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:57.991190 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:57.991039 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtwj\" (UniqueName: \"kubernetes.io/projected/07cf35ad-8c1c-4291-989c-fefeecf83efe-kube-api-access-whtwj\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:40:58.205570 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:58.205543 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l"] Jul 10 19:40:58.211850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:58.211828 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-43c766/test-model-43c766-kserve-7f67667cbc-9465l"] Jul 10 19:40:59.071189 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:59.071162 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07cf35ad-8c1c-4291-989c-fefeecf83efe" path="/var/lib/kubelet/pods/07cf35ad-8c1c-4291-989c-fefeecf83efe/volumes" Jul 10 19:40:59.071553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:40:59.071539 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84cd1af-f705-4e66-8487-74b79c334f6c" path="/var/lib/kubelet/pods/b84cd1af-f705-4e66-8487-74b79c334f6c/volumes" Jul 10 19:41:24.899015 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:24.898977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:25.156251 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:25.156174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:25.378316 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:25.378284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:25.547729 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:25.547636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:25.996014 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:25.995977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:26.102299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.102267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:26.323886 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.323797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:26.324977 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.324956 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv"] Jul 10 19:41:26.325180 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.325152 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" podUID="d715453e-fb6a-49db-86bd-35e99c1befa1" containerName="maas-api" containerID="cri-o://319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587" gracePeriod=30 Jul 10 19:41:26.377815 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.377789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:26.453970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.453937 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" podUID="d715453e-fb6a-49db-86bd-35e99c1befa1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.43:8443/health\": dial tcp 10.133.0.43:8443: connect: connection refused" Jul 10 19:41:26.768596 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.768572 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:41:26.824835 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.824811 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls\") pod \"d715453e-fb6a-49db-86bd-35e99c1befa1\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " Jul 10 19:41:26.824973 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.824899 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d715453e-fb6a-49db-86bd-35e99c1befa1-openshift-service-ca\") pod \"d715453e-fb6a-49db-86bd-35e99c1befa1\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " Jul 10 19:41:26.824973 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.824923 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv2cg\" (UniqueName: \"kubernetes.io/projected/d715453e-fb6a-49db-86bd-35e99c1befa1-kube-api-access-hv2cg\") pod \"d715453e-fb6a-49db-86bd-35e99c1befa1\" (UID: \"d715453e-fb6a-49db-86bd-35e99c1befa1\") " Jul 10 19:41:26.825253 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.825229 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d715453e-fb6a-49db-86bd-35e99c1befa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d715453e-fb6a-49db-86bd-35e99c1befa1" (UID: "d715453e-fb6a-49db-86bd-35e99c1befa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:41:26.826870 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.826848 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d715453e-fb6a-49db-86bd-35e99c1befa1" (UID: "d715453e-fb6a-49db-86bd-35e99c1befa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:41:26.826992 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.826973 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d715453e-fb6a-49db-86bd-35e99c1befa1-kube-api-access-hv2cg" (OuterVolumeSpecName: "kube-api-access-hv2cg") pod "d715453e-fb6a-49db-86bd-35e99c1befa1" (UID: "d715453e-fb6a-49db-86bd-35e99c1befa1"). InnerVolumeSpecName "kube-api-access-hv2cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:41:26.925842 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.925770 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d715453e-fb6a-49db-86bd-35e99c1befa1-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:41:26.925842 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.925804 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d715453e-fb6a-49db-86bd-35e99c1befa1-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:41:26.925842 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:26.925819 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2cg\" (UniqueName: \"kubernetes.io/projected/d715453e-fb6a-49db-86bd-35e99c1befa1-kube-api-access-hv2cg\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:41:27.027237 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.027213 2579 generic.go:356] "Generic (PLEG): container finished" podID="d715453e-fb6a-49db-86bd-35e99c1befa1" containerID="319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587" exitCode=0 Jul 10 19:41:27.027531 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.027257 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" Jul 10 19:41:27.027531 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.027305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" event={"ID":"d715453e-fb6a-49db-86bd-35e99c1befa1","Type":"ContainerDied","Data":"319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587"} Jul 10 19:41:27.027531 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.027360 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv" event={"ID":"d715453e-fb6a-49db-86bd-35e99c1befa1","Type":"ContainerDied","Data":"8cdfe6e43fb6220d4f5b9fcabcc6138ea366125b2c33eeb136b472c8166283e9"} Jul 10 19:41:27.027531 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.027383 2579 scope.go:117] "RemoveContainer" containerID="319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587" Jul 10 19:41:27.036421 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.036407 2579 scope.go:117] "RemoveContainer" containerID="319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587" Jul 10 19:41:27.036675 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:41:27.036641 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587\": container with ID starting with 319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587 not found: ID does not exist" containerID="319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587" Jul 10 19:41:27.036741 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.036679 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587"} err="failed to get container status \"319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587\": rpc error: code = NotFound desc = could not find container \"319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587\": container with ID starting with 319af9c8a5eede9c8d4fb128c34f9ecfbcdf333b8b4375078740ab720c4f1587 not found: ID does not exist" Jul 10 19:41:27.048997 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.048974 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv"] Jul 10 19:41:27.053220 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.053201 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-43c766-78d5756b46-kw6mv"] Jul 10 19:41:27.070449 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:27.070423 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d715453e-fb6a-49db-86bd-35e99c1befa1" path="/var/lib/kubelet/pods/d715453e-fb6a-49db-86bd-35e99c1befa1/volumes" Jul 10 19:41:31.581526 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:31.581489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:31.681863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:31.681830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:31.844337 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:31.844307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:32.061242 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:32.061210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:32.392695 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:32.392640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:32.695135 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:32.695049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:33.047004 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:33.046926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:35.956263 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:35.956231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:36.060953 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.060925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:36.068726 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.068701 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9"] Jul 10 19:41:36.069100 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069087 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07cf35ad-8c1c-4291-989c-fefeecf83efe" containerName="main" Jul 10 19:41:36.069100 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069101 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="07cf35ad-8c1c-4291-989c-fefeecf83efe" containerName="main" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069115 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d715453e-fb6a-49db-86bd-35e99c1befa1" containerName="maas-api" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069121 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d715453e-fb6a-49db-86bd-35e99c1befa1" containerName="maas-api" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069130 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2180e11d-b566-44e0-89a7-c2695bc70b3c" containerName="maas-api" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069139 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2180e11d-b566-44e0-89a7-c2695bc70b3c" containerName="maas-api" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069149 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84cd1af-f705-4e66-8487-74b79c334f6c" containerName="main" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069154 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84cd1af-f705-4e66-8487-74b79c334f6c" containerName="main" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069216 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="07cf35ad-8c1c-4291-989c-fefeecf83efe" containerName="main" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069229 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2180e11d-b566-44e0-89a7-c2695bc70b3c" containerName="maas-api" Jul 10 19:41:36.069232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069236 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b84cd1af-f705-4e66-8487-74b79c334f6c" containerName="main" Jul 10 19:41:36.069718 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.069244 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d715453e-fb6a-49db-86bd-35e99c1befa1" containerName="maas-api" Jul 10 19:41:36.073608 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.073574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.076312 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.076294 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2fa627-serving-cert\"" Jul 10 19:41:36.080634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.080613 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9"] Jul 10 19:41:36.156389 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.156357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:36.203566 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.203540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.203691 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.203575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-openshift-service-ca\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.203691 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.203600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvlw\" (UniqueName: \"kubernetes.io/projected/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-kube-api-access-8nvlw\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.305062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.304967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.305062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.305011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-openshift-service-ca\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.305062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.305030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvlw\" (UniqueName: \"kubernetes.io/projected/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-kube-api-access-8nvlw\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.305290 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:41:36.305169 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-serving-cert: secret "maas-api-e2e-api-a-2fa627-serving-cert" not found Jul 10 19:41:36.305290 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:41:36.305251 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls podName:45897f2a-b60a-4497-8e7d-d0ce5ae7f078 nodeName:}" failed. No retries permitted until 2026-07-10 19:41:36.805234831 +0000 UTC m=+2036.844107825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls") pod "maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" (UID: "45897f2a-b60a-4497-8e7d-d0ce5ae7f078") : secret "maas-api-e2e-api-a-2fa627-serving-cert" not found Jul 10 19:41:36.305568 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.305547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-openshift-service-ca\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.319314 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.319292 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvlw\" (UniqueName: \"kubernetes.io/projected/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-kube-api-access-8nvlw\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.667933 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.667905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:36.766878 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.766848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:36.809171 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.809138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.811454 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.811433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls\") pod \"maas-api-e2e-api-a-2fa627-854c74589c-fq7m9\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:36.835393 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.835374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:36.986846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:36.986772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:37.122495 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:37.122234 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9"] Jul 10 19:41:38.082999 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:38.082911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" event={"ID":"45897f2a-b60a-4497-8e7d-d0ce5ae7f078","Type":"ContainerStarted","Data":"7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978"} Jul 10 19:41:38.082999 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:38.082964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" event={"ID":"45897f2a-b60a-4497-8e7d-d0ce5ae7f078","Type":"ContainerStarted","Data":"6d31766aca5a854ac96cf92047670c0bb7a5fdb7a2c1b8d790a7e46f6e0d1360"} Jul 10 19:41:38.083406 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:38.083042 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:38.101388 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:38.101339 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" podStartSLOduration=1.413955059 podStartE2EDuration="2.10132314s" totalImagesPullingTime="687.368081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:41:38.100001488 +0000 UTC m=+2038.138874524" watchObservedRunningTime="2026-07-10 19:41:38.10132314 +0000 UTC m=+2038.140196157" Jul 10 19:41:40.420437 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:40.420396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:40.854957 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:40.854928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:41.160371 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:41.160280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:41.275814 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:41.275784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:44.092138 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.092106 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:44.107028 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.106987 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k"] Jul 10 19:41:44.111878 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.111404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.118439 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.118417 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b3f5d9-serving-cert\"" Jul 10 19:41:44.124888 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.124863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k"] Jul 10 19:41:44.230977 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.230951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:44.275892 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.275868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59fda1-584d-4eab-97f8-bcec429a23f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.275984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.275903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgqs\" (UniqueName: \"kubernetes.io/projected/5e59fda1-584d-4eab-97f8-bcec429a23f2-kube-api-access-mmgqs\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.275984 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.275955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.333446 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.333420 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:44.376875 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.376822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59fda1-584d-4eab-97f8-bcec429a23f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.376875 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.376856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgqs\" (UniqueName: \"kubernetes.io/projected/5e59fda1-584d-4eab-97f8-bcec429a23f2-kube-api-access-mmgqs\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.377030 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.376877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.377030 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:41:44.376995 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-serving-cert: secret "maas-api-e2e-api-b-b3f5d9-serving-cert" not found Jul 10 19:41:44.377131 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:41:44.377043 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls podName:5e59fda1-584d-4eab-97f8-bcec429a23f2 nodeName:}" failed. No retries permitted until 2026-07-10 19:41:44.87702758 +0000 UTC m=+2044.915900575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls") pod "maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" (UID: "5e59fda1-584d-4eab-97f8-bcec429a23f2") : secret "maas-api-e2e-api-b-b3f5d9-serving-cert" not found Jul 10 19:41:44.377422 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.377404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59fda1-584d-4eab-97f8-bcec429a23f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.388293 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.388268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgqs\" (UniqueName: \"kubernetes.io/projected/5e59fda1-584d-4eab-97f8-bcec429a23f2-kube-api-access-mmgqs\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.756224 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.756150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:44.856822 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.856788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:44.881016 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.880993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:44.883392 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:44.883369 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls\") pod \"maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:45.033274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:45.033192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:45.163821 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:45.163798 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k"] Jul 10 19:41:45.165765 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:41:45.165738 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e59fda1_584d_4eab_97f8_bcec429a23f2.slice/crio-3f69034b626fe586475b644cdfa904dcb21c8a309ae6bba4389fb619a225187a WatchSource:0}: Error finding container 3f69034b626fe586475b644cdfa904dcb21c8a309ae6bba4389fb619a225187a: Status 404 returned error can't find the container with id 3f69034b626fe586475b644cdfa904dcb21c8a309ae6bba4389fb619a225187a Jul 10 19:41:46.134244 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:46.134208 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" event={"ID":"5e59fda1-584d-4eab-97f8-bcec429a23f2","Type":"ContainerStarted","Data":"5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8"} Jul 10 19:41:46.134412 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:46.134266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" event={"ID":"5e59fda1-584d-4eab-97f8-bcec429a23f2","Type":"ContainerStarted","Data":"3f69034b626fe586475b644cdfa904dcb21c8a309ae6bba4389fb619a225187a"} Jul 10 19:41:46.134412 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:46.134312 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:46.151442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:46.151392 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" podStartSLOduration=1.776678273 podStartE2EDuration="2.151379961s" totalImagesPullingTime="374.701688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:41:46.150017916 +0000 UTC m=+2046.188890980" watchObservedRunningTime="2026-07-10 19:41:46.151379961 +0000 UTC m=+2046.190252977" Jul 10 19:41:48.524313 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:48.524279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:52.144726 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:52.144701 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:41:56.480824 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:56.480777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:56.917468 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:56.917438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:57.068084 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.068052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:57.168678 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.168587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:57.302043 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.302013 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9"] Jul 10 19:41:57.302249 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.302227 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" podUID="45897f2a-b60a-4497-8e7d-d0ce5ae7f078" containerName="maas-api" containerID="cri-o://7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978" gracePeriod=30 Jul 10 19:41:57.336575 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.336543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:57.551046 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.551025 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:57.589284 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.589259 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nvlw\" (UniqueName: \"kubernetes.io/projected/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-kube-api-access-8nvlw\") pod \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " Jul 10 19:41:57.589398 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.589288 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-openshift-service-ca\") pod \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " Jul 10 19:41:57.589398 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.589329 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls\") pod \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\" (UID: \"45897f2a-b60a-4497-8e7d-d0ce5ae7f078\") " Jul 10 19:41:57.589615 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.589596 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45897f2a-b60a-4497-8e7d-d0ce5ae7f078" (UID: "45897f2a-b60a-4497-8e7d-d0ce5ae7f078"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:41:57.589728 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.589709 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:41:57.591269 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.591245 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-kube-api-access-8nvlw" (OuterVolumeSpecName: "kube-api-access-8nvlw") pod "45897f2a-b60a-4497-8e7d-d0ce5ae7f078" (UID: "45897f2a-b60a-4497-8e7d-d0ce5ae7f078"). InnerVolumeSpecName "kube-api-access-8nvlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:41:57.591380 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.591364 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45897f2a-b60a-4497-8e7d-d0ce5ae7f078" (UID: "45897f2a-b60a-4497-8e7d-d0ce5ae7f078"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:41:57.690901 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.690851 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nvlw\" (UniqueName: \"kubernetes.io/projected/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-kube-api-access-8nvlw\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:41:57.690901 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:57.690871 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45897f2a-b60a-4497-8e7d-d0ce5ae7f078-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:41:58.175456 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.175424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:58.191325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.191297 2579 generic.go:356] "Generic (PLEG): container finished" podID="45897f2a-b60a-4497-8e7d-d0ce5ae7f078" containerID="7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978" exitCode=0 Jul 10 19:41:58.191470 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.191348 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" Jul 10 19:41:58.191470 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.191348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" event={"ID":"45897f2a-b60a-4497-8e7d-d0ce5ae7f078","Type":"ContainerDied","Data":"7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978"} Jul 10 19:41:58.191586 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.191469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9" event={"ID":"45897f2a-b60a-4497-8e7d-d0ce5ae7f078","Type":"ContainerDied","Data":"6d31766aca5a854ac96cf92047670c0bb7a5fdb7a2c1b8d790a7e46f6e0d1360"} Jul 10 19:41:58.191586 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.191493 2579 scope.go:117] "RemoveContainer" containerID="7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978" Jul 10 19:41:58.200215 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.200199 2579 scope.go:117] "RemoveContainer" containerID="7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978" Jul 10 19:41:58.200455 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:41:58.200439 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978\": container with ID starting with 7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978 not found: ID does not exist" containerID="7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978" Jul 10 19:41:58.200503 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.200463 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978"} err="failed to get container status \"7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978\": rpc error: code = NotFound desc = could not find container \"7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978\": container with ID starting with 7f5c20df1a016ab7f61dd30eda3d06a145b502deb84f8d4666d855e8eed2d978 not found: ID does not exist" Jul 10 19:41:58.214415 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.214392 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9"] Jul 10 19:41:58.219975 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.219953 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2fa627-854c74589c-fq7m9"] Jul 10 19:41:58.230092 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:58.230073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:59.071361 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:41:59.071333 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45897f2a-b60a-4497-8e7d-d0ce5ae7f078" path="/var/lib/kubelet/pods/45897f2a-b60a-4497-8e7d-d0ce5ae7f078/volumes" Jul 10 19:42:02.789179 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:02.789094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:02.850855 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:02.850821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:03.042515 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:03.042434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:03.261032 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:03.261003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:03.491628 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:03.491595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:03.795459 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:03.795371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:03.961145 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:03.961111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:04.062825 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.062748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:04.211438 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.211408 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k"] Jul 10 19:42:04.211666 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.211630 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" podUID="5e59fda1-584d-4eab-97f8-bcec429a23f2" containerName="maas-api" containerID="cri-o://5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8" gracePeriod=30 Jul 10 19:42:04.254172 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.254139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:04.457434 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.457412 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:42:04.551164 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.551138 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59fda1-584d-4eab-97f8-bcec429a23f2-openshift-service-ca\") pod \"5e59fda1-584d-4eab-97f8-bcec429a23f2\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " Jul 10 19:42:04.551305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.551195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls\") pod \"5e59fda1-584d-4eab-97f8-bcec429a23f2\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " Jul 10 19:42:04.551305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.551235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmgqs\" (UniqueName: \"kubernetes.io/projected/5e59fda1-584d-4eab-97f8-bcec429a23f2-kube-api-access-mmgqs\") pod \"5e59fda1-584d-4eab-97f8-bcec429a23f2\" (UID: \"5e59fda1-584d-4eab-97f8-bcec429a23f2\") " Jul 10 19:42:04.551524 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.551499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e59fda1-584d-4eab-97f8-bcec429a23f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e59fda1-584d-4eab-97f8-bcec429a23f2" (UID: "5e59fda1-584d-4eab-97f8-bcec429a23f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:42:04.553157 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.553138 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e59fda1-584d-4eab-97f8-bcec429a23f2" (UID: "5e59fda1-584d-4eab-97f8-bcec429a23f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:42:04.553228 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.553187 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e59fda1-584d-4eab-97f8-bcec429a23f2-kube-api-access-mmgqs" (OuterVolumeSpecName: "kube-api-access-mmgqs") pod "5e59fda1-584d-4eab-97f8-bcec429a23f2" (UID: "5e59fda1-584d-4eab-97f8-bcec429a23f2"). InnerVolumeSpecName "kube-api-access-mmgqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:42:04.652827 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.652799 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e59fda1-584d-4eab-97f8-bcec429a23f2-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:42:04.652827 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.652825 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e59fda1-584d-4eab-97f8-bcec429a23f2-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:42:04.652971 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:04.652834 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmgqs\" (UniqueName: \"kubernetes.io/projected/5e59fda1-584d-4eab-97f8-bcec429a23f2-kube-api-access-mmgqs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:42:05.227000 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.226967 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e59fda1-584d-4eab-97f8-bcec429a23f2" containerID="5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8" exitCode=0 Jul 10 19:42:05.227358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.227014 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" Jul 10 19:42:05.227358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.227058 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" event={"ID":"5e59fda1-584d-4eab-97f8-bcec429a23f2","Type":"ContainerDied","Data":"5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8"} Jul 10 19:42:05.227358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.227110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k" event={"ID":"5e59fda1-584d-4eab-97f8-bcec429a23f2","Type":"ContainerDied","Data":"3f69034b626fe586475b644cdfa904dcb21c8a309ae6bba4389fb619a225187a"} Jul 10 19:42:05.227358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.227138 2579 scope.go:117] "RemoveContainer" containerID="5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8" Jul 10 19:42:05.235453 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.235440 2579 scope.go:117] "RemoveContainer" containerID="5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8" Jul 10 19:42:05.235765 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:42:05.235737 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8\": container with ID starting with 5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8 not found: ID does not exist" containerID="5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8" Jul 10 19:42:05.235863 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.235773 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8"} err="failed to get container status \"5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8\": rpc error: code = NotFound desc = could not find container \"5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8\": container with ID starting with 5e6269f93ffcff0ed7f6c61312ac9b2fe1ac40ba6da3b049607647e9cdfefac8 not found: ID does not exist" Jul 10 19:42:05.244451 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.244428 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k"] Jul 10 19:42:05.250666 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:05.250636 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3f5d9-6f97b7f89f-drg5k"] Jul 10 19:42:07.071147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:07.071115 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e59fda1-584d-4eab-97f8-bcec429a23f2" path="/var/lib/kubelet/pods/5e59fda1-584d-4eab-97f8-bcec429a23f2/volumes" Jul 10 19:42:09.065721 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:09.065684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:09.166575 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:09.166545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:09.668287 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:09.668255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:09.735090 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:09.735045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:09.931833 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:09.931754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:10.137278 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:10.137241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:10.452967 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:10.452935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:10.771888 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:10.771797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:10.918833 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:10.918806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:13.154276 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.154244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:13.268682 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.260794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:13.271556 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.271531 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg"] Jul 10 19:42:13.271937 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.271922 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e59fda1-584d-4eab-97f8-bcec429a23f2" containerName="maas-api" Jul 10 19:42:13.271937 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.271938 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e59fda1-584d-4eab-97f8-bcec429a23f2" containerName="maas-api" Jul 10 19:42:13.272037 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.271963 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45897f2a-b60a-4497-8e7d-d0ce5ae7f078" containerName="maas-api" Jul 10 19:42:13.272037 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.271968 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45897f2a-b60a-4497-8e7d-d0ce5ae7f078" containerName="maas-api" Jul 10 19:42:13.272037 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.272029 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e59fda1-584d-4eab-97f8-bcec429a23f2" containerName="maas-api" Jul 10 19:42:13.272131 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.272039 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45897f2a-b60a-4497-8e7d-d0ce5ae7f078" containerName="maas-api" Jul 10 19:42:13.282202 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.282178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg"] Jul 10 19:42:13.282342 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.282277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.285754 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.285730 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f25aa8-serving-cert\"" Jul 10 19:42:13.325020 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.324992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvxj\" (UniqueName: \"kubernetes.io/projected/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-kube-api-access-hcvxj\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.325133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.325030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.325133 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.325069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.356825 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.356797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:13.425631 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.425575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.425631 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.425624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.425813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.425719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvxj\" (UniqueName: \"kubernetes.io/projected/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-kube-api-access-hcvxj\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.426261 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.426239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.428036 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.428016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.439310 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.439290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvxj\" (UniqueName: \"kubernetes.io/projected/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-kube-api-access-hcvxj\") pod \"maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.457585 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.457562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:13.592814 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.592791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:13.723880 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.723851 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg"] Jul 10 19:42:13.726010 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:42:13.725971 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bb8f87_96ec_4ed4_b58a_c8b6663819b7.slice/crio-c457d16713d8d1a903aa6fea065dd3acd3c537c8b3660a4a5ed8d6830f8549d6 WatchSource:0}: Error finding container c457d16713d8d1a903aa6fea065dd3acd3c537c8b3660a4a5ed8d6830f8549d6: Status 404 returned error can't find the container with id c457d16713d8d1a903aa6fea065dd3acd3c537c8b3660a4a5ed8d6830f8549d6 Jul 10 19:42:13.811332 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.811306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:13.912771 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:13.912734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:14.279062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:14.279031 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" event={"ID":"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7","Type":"ContainerStarted","Data":"1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa"} Jul 10 19:42:14.279435 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:14.279079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" event={"ID":"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7","Type":"ContainerStarted","Data":"c457d16713d8d1a903aa6fea065dd3acd3c537c8b3660a4a5ed8d6830f8549d6"} Jul 10 19:42:14.279435 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:14.279100 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:14.298343 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:14.298299 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" podStartSLOduration=0.892808387 podStartE2EDuration="1.298284887s" totalImagesPullingTime="405.4765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:42:14.296346034 +0000 UTC m=+2074.335219052" watchObservedRunningTime="2026-07-10 19:42:14.298284887 +0000 UTC m=+2074.337157940" Jul 10 19:42:17.657632 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:17.657600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:17.974184 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:17.974102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:18.286954 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:18.286875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:18.591238 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:18.591209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:20.289616 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:20.289586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:42:21.213519 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.213482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:21.286153 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.286123 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf"] Jul 10 19:42:21.289639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.289616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.292332 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.292314 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-fc4b18-serving-cert\"" Jul 10 19:42:21.302795 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.302773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf"] Jul 10 19:42:21.306032 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.306011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:21.374768 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.374741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:21.393670 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.393631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx6lb\" (UniqueName: \"kubernetes.io/projected/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-kube-api-access-vx6lb\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.393767 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.393723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.393808 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.393773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.495267 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.495185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx6lb\" (UniqueName: \"kubernetes.io/projected/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-kube-api-access-vx6lb\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.495505 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.495284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.495505 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.495333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.495505 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:42:21.495400 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-serving-cert: secret "maas-api-e2e-shared-b-fc4b18-serving-cert" not found Jul 10 19:42:21.495505 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:42:21.495456 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls podName:8d3af1ea-1bb4-4742-adf6-c6550980c8d8 nodeName:}" failed. No retries permitted until 2026-07-10 19:42:21.995441131 +0000 UTC m=+2082.034314126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls") pod "maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" (UID: "8d3af1ea-1bb4-4742-adf6-c6550980c8d8") : secret "maas-api-e2e-shared-b-fc4b18-serving-cert" not found Jul 10 19:42:21.495937 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.495920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.504077 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.504058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx6lb\" (UniqueName: \"kubernetes.io/projected/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-kube-api-access-vx6lb\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:21.621156 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.621125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:21.750245 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.750153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:21.850866 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:21.850837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:22.000300 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:22.000213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:22.002637 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:22.002613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:22.200600 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:22.200573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:22.327277 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:22.327251 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf"] Jul 10 19:42:22.329008 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:42:22.328980 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3af1ea_1bb4_4742_adf6_c6550980c8d8.slice/crio-ca18a8f060db35e74edb1534a7b411483da0ca14c383bcd9b3415bc818b4b206 WatchSource:0}: Error finding container ca18a8f060db35e74edb1534a7b411483da0ca14c383bcd9b3415bc818b4b206: Status 404 returned error can't find the container with id ca18a8f060db35e74edb1534a7b411483da0ca14c383bcd9b3415bc818b4b206 Jul 10 19:42:23.323261 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:23.323172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" event={"ID":"8d3af1ea-1bb4-4742-adf6-c6550980c8d8","Type":"ContainerStarted","Data":"3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667"} Jul 10 19:42:23.323261 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:23.323227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" event={"ID":"8d3af1ea-1bb4-4742-adf6-c6550980c8d8","Type":"ContainerStarted","Data":"ca18a8f060db35e74edb1534a7b411483da0ca14c383bcd9b3415bc818b4b206"} Jul 10 19:42:23.323478 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:23.323317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:23.342882 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:23.342826 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" podStartSLOduration=1.643495337 podStartE2EDuration="2.342807758s" totalImagesPullingTime="699.312421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:42:23.341083187 +0000 UTC m=+2083.379956207" watchObservedRunningTime="2026-07-10 19:42:23.342807758 +0000 UTC m=+2083.381680777" Jul 10 19:42:25.686455 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:25.686419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:29.332919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:29.332892 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:42:34.552339 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:34.552308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:34.653038 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:34.653006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:37.770313 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.770286 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k"] Jul 10 19:42:37.776271 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.776251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:37.779087 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.779065 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f25aa8\"/\"openshift-service-ca.crt\"" Jul 10 19:42:37.780678 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.780639 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f25aa8\"/\"auth-test-model-f25aa8-kserve-self-signed-certs\"" Jul 10 19:42:37.781069 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.780878 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f25aa8\"/\"default-dockercfg-7jgqb\"" Jul 10 19:42:37.781069 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.780912 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f25aa8\"/\"kube-root-ca.crt\"" Jul 10 19:42:37.783255 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.783228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k"] Jul 10 19:42:37.799553 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.799529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:37.937413 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.937380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-model-cache\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:37.937590 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.937434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c44fac5-a83f-41ce-a047-a74063e1fffb-tls-certs\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:37.937590 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.937479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwp9\" (UniqueName: \"kubernetes.io/projected/3c44fac5-a83f-41ce-a047-a74063e1fffb-kube-api-access-9lwp9\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:37.937590 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.937524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-dshm\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:37.937590 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.937575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-tmp-dir\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:37.937773 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:37.937695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-home\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038194 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-tmp-dir\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038194 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-home\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038390 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-model-cache\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038390 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c44fac5-a83f-41ce-a047-a74063e1fffb-tls-certs\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038390 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwp9\" (UniqueName: \"kubernetes.io/projected/3c44fac5-a83f-41ce-a047-a74063e1fffb-kube-api-access-9lwp9\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038538 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-dshm\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038591 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038535 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-tmp-dir\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038681 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038612 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-home\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.038723 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.038675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-model-cache\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.040540 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.040519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-dshm\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.040717 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.040700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c44fac5-a83f-41ce-a047-a74063e1fffb-tls-certs\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.054107 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.054083 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwp9\" (UniqueName: \"kubernetes.io/projected/3c44fac5-a83f-41ce-a047-a74063e1fffb-kube-api-access-9lwp9\") pod \"auth-test-model-f25aa8-kserve-5556bb5f-6nn8k\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.086126 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.086101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:38.218502 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.218479 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k"] Jul 10 19:42:38.220596 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:42:38.220567 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c44fac5_a83f_41ce_a047_a74063e1fffb.slice/crio-84fdd14158cf3e592911a4dcbf7679310319613c78baadb2fcf73526e911777e WatchSource:0}: Error finding container 84fdd14158cf3e592911a4dcbf7679310319613c78baadb2fcf73526e911777e: Status 404 returned error can't find the container with id 84fdd14158cf3e592911a4dcbf7679310319613c78baadb2fcf73526e911777e Jul 10 19:42:38.403970 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.403939 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" event={"ID":"3c44fac5-a83f-41ce-a047-a74063e1fffb","Type":"ContainerStarted","Data":"c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1"} Jul 10 19:42:38.404112 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.403980 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" event={"ID":"3c44fac5-a83f-41ce-a047-a74063e1fffb","Type":"ContainerStarted","Data":"84fdd14158cf3e592911a4dcbf7679310319613c78baadb2fcf73526e911777e"} Jul 10 19:42:38.442761 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:38.442714 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" podStartSLOduration=1.442697839 podStartE2EDuration="1.442697839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:42:38.44182718 +0000 UTC m=+2098.480700198" watchObservedRunningTime="2026-07-10 19:42:38.442697839 +0000 UTC m=+2098.481570855" Jul 10 19:42:41.042330 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:41.042298 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:42:41.049773 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:41.049749 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:42:42.645811 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:42.645781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:42.770907 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:42.770872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:46.452781 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:46.452751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:48.086331 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:48.086304 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:48.086702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:48.086348 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:48.098876 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:48.098853 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:48.463530 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:48.463461 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:42:57.375688 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.375639 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr"] Jul 10 19:42:57.379280 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.379262 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.382067 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.382044 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"openshift-service-ca.crt\"" Jul 10 19:42:57.383269 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.383249 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"default-dockercfg-988nc\"" Jul 10 19:42:57.383377 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.383317 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"kube-root-ca.crt\"" Jul 10 19:42:57.383489 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.383456 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"auth-test-model-fc4b18-kserve-self-signed-certs\"" Jul 10 19:42:57.386615 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.386296 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr"] Jul 10 19:42:57.402045 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.402019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:57.497481 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.497454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-model-cache\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.497581 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.497503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-dshm\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.497680 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.497595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mwc\" (UniqueName: \"kubernetes.io/projected/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-kube-api-access-x6mwc\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.497680 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.497638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tls-certs\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.497800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.497688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tmp-dir\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.497800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.497719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-home\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.598434 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.598413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-dshm\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.598528 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.598461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mwc\" (UniqueName: \"kubernetes.io/projected/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-kube-api-access-x6mwc\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.598613 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.598591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tls-certs\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.598696 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.598634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tmp-dir\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.598696 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.598686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-home\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.598816 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.598768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-model-cache\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.598993 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.598971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tmp-dir\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.599075 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.599023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-home\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.599124 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.599073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-model-cache\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.600680 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.600647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-dshm\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.600962 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.600946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tls-certs\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.610469 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.610444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mwc\" (UniqueName: \"kubernetes.io/projected/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-kube-api-access-x6mwc\") pod \"auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.688634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.688582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:42:57.816313 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:57.816274 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr"] Jul 10 19:42:57.819173 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:42:57.819134 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a2f9f2_3438_4ada_ae2b_78c15f08a377.slice/crio-46b985cd658db6193991cd8bb75ce37269c418b9f5632ac18d86d888e5031e50 WatchSource:0}: Error finding container 46b985cd658db6193991cd8bb75ce37269c418b9f5632ac18d86d888e5031e50: Status 404 returned error can't find the container with id 46b985cd658db6193991cd8bb75ce37269c418b9f5632ac18d86d888e5031e50 Jul 10 19:42:58.510699 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:58.510662 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" event={"ID":"e8a2f9f2-3438-4ada-ae2b-78c15f08a377","Type":"ContainerStarted","Data":"0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d"} Jul 10 19:42:58.511105 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:58.510706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" event={"ID":"e8a2f9f2-3438-4ada-ae2b-78c15f08a377","Type":"ContainerStarted","Data":"46b985cd658db6193991cd8bb75ce37269c418b9f5632ac18d86d888e5031e50"} Jul 10 19:42:58.529946 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:42:58.529894 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" podStartSLOduration=1.529880336 podStartE2EDuration="1.529880336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:42:58.52817905 +0000 UTC m=+2118.567052067" watchObservedRunningTime="2026-07-10 19:42:58.529880336 +0000 UTC m=+2118.568753394" Jul 10 19:43:07.689233 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:43:07.689199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:43:07.689619 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:43:07.689251 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:43:07.701190 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:43:07.701158 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:43:08.569689 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:43:08.569644 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:43:20.325611 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:43:20.325574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:43:20.423739 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:43:20.423705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:28.567950 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:28.567920 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k"] Jul 10 19:44:28.590072 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:28.590043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:29.690836 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:29.690805 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr"] Jul 10 19:44:30.189299 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:30.189268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:33.678634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.678603 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g"] Jul 10 19:44:33.682648 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.682631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.687027 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.687009 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f25aa8\"/\"sub-test-model-f25aa8-kserve-self-signed-certs\"" Jul 10 19:44:33.699961 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.699943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g"] Jul 10 19:44:33.721427 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.721407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-model-cache\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.721523 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.721439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tls-certs\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.721523 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.721471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-home\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.721598 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.721578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5ms\" (UniqueName: \"kubernetes.io/projected/729c7370-4e8e-4a5c-bf25-2581fa218bfd-kube-api-access-9m5ms\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.721633 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.721619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tmp-dir\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.721724 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.721707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-dshm\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.724386 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.724359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:33.822620 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.822596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-dshm\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.822734 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.822634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-model-cache\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.822785 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.822769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tls-certs\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.822849 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.822831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-home\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.822920 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.822903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5ms\" (UniqueName: \"kubernetes.io/projected/729c7370-4e8e-4a5c-bf25-2581fa218bfd-kube-api-access-9m5ms\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.822979 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.822962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tmp-dir\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.823086 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.823066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-model-cache\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.823149 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.823088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-home\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.823271 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.823248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tmp-dir\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.824901 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.824885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-dshm\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.825252 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.825232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tls-certs\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.832282 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.832264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5ms\" (UniqueName: \"kubernetes.io/projected/729c7370-4e8e-4a5c-bf25-2581fa218bfd-kube-api-access-9m5ms\") pod \"sub-test-model-f25aa8-kserve-699d4bff69-47g7g\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:33.991414 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:33.991342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:34.129998 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:34.129973 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g"] Jul 10 19:44:34.131944 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:44:34.131908 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod729c7370_4e8e_4a5c_bf25_2581fa218bfd.slice/crio-55b6d32c32fa89c6f53eb693646bdb26ca5d816aacdf0e234ff79e60d3a2977a WatchSource:0}: Error finding container 55b6d32c32fa89c6f53eb693646bdb26ca5d816aacdf0e234ff79e60d3a2977a: Status 404 returned error can't find the container with id 55b6d32c32fa89c6f53eb693646bdb26ca5d816aacdf0e234ff79e60d3a2977a Jul 10 19:44:34.992113 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:34.992079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" event={"ID":"729c7370-4e8e-4a5c-bf25-2581fa218bfd","Type":"ContainerStarted","Data":"896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50"} Jul 10 19:44:34.992476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:34.992119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" event={"ID":"729c7370-4e8e-4a5c-bf25-2581fa218bfd","Type":"ContainerStarted","Data":"55b6d32c32fa89c6f53eb693646bdb26ca5d816aacdf0e234ff79e60d3a2977a"} Jul 10 19:44:35.011414 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:35.011363 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" podStartSLOduration=2.011347969 podStartE2EDuration="2.011347969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:44:35.010270643 +0000 UTC m=+2215.049143692" watchObservedRunningTime="2026-07-10 19:44:35.011347969 +0000 UTC m=+2215.050221036" Jul 10 19:44:43.588356 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.588313 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" podUID="3c44fac5-a83f-41ce-a047-a74063e1fffb" containerName="main" containerID="cri-o://c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1" gracePeriod=45 Jul 10 19:44:43.837618 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.837595 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:44:43.919945 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.919877 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c44fac5-a83f-41ce-a047-a74063e1fffb-tls-certs\") pod \"3c44fac5-a83f-41ce-a047-a74063e1fffb\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " Jul 10 19:44:43.920072 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.919948 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-tmp-dir\") pod \"3c44fac5-a83f-41ce-a047-a74063e1fffb\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " Jul 10 19:44:43.920072 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920005 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-dshm\") pod \"3c44fac5-a83f-41ce-a047-a74063e1fffb\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " Jul 10 19:44:43.920072 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920046 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-model-cache\") pod \"3c44fac5-a83f-41ce-a047-a74063e1fffb\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " Jul 10 19:44:43.920224 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920099 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-home\") pod \"3c44fac5-a83f-41ce-a047-a74063e1fffb\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " Jul 10 19:44:43.920224 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920138 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lwp9\" (UniqueName: \"kubernetes.io/projected/3c44fac5-a83f-41ce-a047-a74063e1fffb-kube-api-access-9lwp9\") pod \"3c44fac5-a83f-41ce-a047-a74063e1fffb\" (UID: \"3c44fac5-a83f-41ce-a047-a74063e1fffb\") " Jul 10 19:44:43.920329 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920277 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c44fac5-a83f-41ce-a047-a74063e1fffb" (UID: "3c44fac5-a83f-41ce-a047-a74063e1fffb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:43.920399 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920373 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c44fac5-a83f-41ce-a047-a74063e1fffb" (UID: "3c44fac5-a83f-41ce-a047-a74063e1fffb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:43.920451 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920386 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-home" (OuterVolumeSpecName: "home") pod "3c44fac5-a83f-41ce-a047-a74063e1fffb" (UID: "3c44fac5-a83f-41ce-a047-a74063e1fffb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:43.920563 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920544 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:43.920616 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920568 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:43.920616 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.920581 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:43.922283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.922256 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-dshm" (OuterVolumeSpecName: "dshm") pod "3c44fac5-a83f-41ce-a047-a74063e1fffb" (UID: "3c44fac5-a83f-41ce-a047-a74063e1fffb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:43.922283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.922267 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c44fac5-a83f-41ce-a047-a74063e1fffb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c44fac5-a83f-41ce-a047-a74063e1fffb" (UID: "3c44fac5-a83f-41ce-a047-a74063e1fffb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:44:43.922283 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.922267 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c44fac5-a83f-41ce-a047-a74063e1fffb-kube-api-access-9lwp9" (OuterVolumeSpecName: "kube-api-access-9lwp9") pod "3c44fac5-a83f-41ce-a047-a74063e1fffb" (UID: "3c44fac5-a83f-41ce-a047-a74063e1fffb"). InnerVolumeSpecName "kube-api-access-9lwp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:43.992150 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.992126 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:43.992247 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:43.992162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:44.004369 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.004346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:44.021924 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.021903 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwp9\" (UniqueName: \"kubernetes.io/projected/3c44fac5-a83f-41ce-a047-a74063e1fffb-kube-api-access-9lwp9\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:44.021924 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.021923 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c44fac5-a83f-41ce-a047-a74063e1fffb-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:44.022032 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.021932 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c44fac5-a83f-41ce-a047-a74063e1fffb-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:44.036866 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.036841 2579 generic.go:356] "Generic (PLEG): container finished" podID="3c44fac5-a83f-41ce-a047-a74063e1fffb" containerID="c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1" exitCode=0 Jul 10 19:44:44.036956 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.036879 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" Jul 10 19:44:44.036956 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.036919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" event={"ID":"3c44fac5-a83f-41ce-a047-a74063e1fffb","Type":"ContainerDied","Data":"c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1"} Jul 10 19:44:44.037049 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.036957 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k" event={"ID":"3c44fac5-a83f-41ce-a047-a74063e1fffb","Type":"ContainerDied","Data":"84fdd14158cf3e592911a4dcbf7679310319613c78baadb2fcf73526e911777e"} Jul 10 19:44:44.037049 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.036980 2579 scope.go:117] "RemoveContainer" containerID="c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1" Jul 10 19:44:44.046039 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.046024 2579 scope.go:117] "RemoveContainer" containerID="c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1" Jul 10 19:44:44.046337 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:44:44.046318 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1\": container with ID starting with c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1 not found: ID does not exist" containerID="c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1" Jul 10 19:44:44.046392 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.046344 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1"} err="failed to get container status \"c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1\": rpc error: code = NotFound desc = could not find container \"c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1\": container with ID starting with c840bacdfc36858719df55eae2d13c27a9292d5d1790e516ce562e0426396ce1 not found: ID does not exist" Jul 10 19:44:44.049065 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.049050 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:44:44.060173 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.060148 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k"] Jul 10 19:44:44.064185 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.064165 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/auth-test-model-f25aa8-kserve-5556bb5f-6nn8k"] Jul 10 19:44:44.710635 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.710590 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" podUID="e8a2f9f2-3438-4ada-ae2b-78c15f08a377" containerName="main" containerID="cri-o://0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d" gracePeriod=45 Jul 10 19:44:44.975252 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:44.975199 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:44:45.032501 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032472 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-home\") pod \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " Jul 10 19:44:45.032624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032533 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tls-certs\") pod \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " Jul 10 19:44:45.032624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032559 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6mwc\" (UniqueName: \"kubernetes.io/projected/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-kube-api-access-x6mwc\") pod \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " Jul 10 19:44:45.032624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tmp-dir\") pod \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " Jul 10 19:44:45.032624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032606 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-model-cache\") pod \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " Jul 10 19:44:45.032835 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032683 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-dshm\") pod \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\" (UID: \"e8a2f9f2-3438-4ada-ae2b-78c15f08a377\") " Jul 10 19:44:45.032835 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032744 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-home" (OuterVolumeSpecName: "home") pod "e8a2f9f2-3438-4ada-ae2b-78c15f08a377" (UID: "e8a2f9f2-3438-4ada-ae2b-78c15f08a377"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:45.032948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032865 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8a2f9f2-3438-4ada-ae2b-78c15f08a377" (UID: "e8a2f9f2-3438-4ada-ae2b-78c15f08a377"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:45.032948 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.032914 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8a2f9f2-3438-4ada-ae2b-78c15f08a377" (UID: "e8a2f9f2-3438-4ada-ae2b-78c15f08a377"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:45.033093 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.033077 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:45.033150 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.033099 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:45.033150 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.033113 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:45.034914 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.034892 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-dshm" (OuterVolumeSpecName: "dshm") pod "e8a2f9f2-3438-4ada-ae2b-78c15f08a377" (UID: "e8a2f9f2-3438-4ada-ae2b-78c15f08a377"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:45.034914 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.034907 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8a2f9f2-3438-4ada-ae2b-78c15f08a377" (UID: "e8a2f9f2-3438-4ada-ae2b-78c15f08a377"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:44:45.035025 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.034908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-kube-api-access-x6mwc" (OuterVolumeSpecName: "kube-api-access-x6mwc") pod "e8a2f9f2-3438-4ada-ae2b-78c15f08a377" (UID: "e8a2f9f2-3438-4ada-ae2b-78c15f08a377"). InnerVolumeSpecName "kube-api-access-x6mwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:45.044613 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.044593 2579 generic.go:356] "Generic (PLEG): container finished" podID="e8a2f9f2-3438-4ada-ae2b-78c15f08a377" containerID="0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d" exitCode=0 Jul 10 19:44:45.044720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.044635 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" Jul 10 19:44:45.044720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.044690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" event={"ID":"e8a2f9f2-3438-4ada-ae2b-78c15f08a377","Type":"ContainerDied","Data":"0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d"} Jul 10 19:44:45.044813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.044734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr" event={"ID":"e8a2f9f2-3438-4ada-ae2b-78c15f08a377","Type":"ContainerDied","Data":"46b985cd658db6193991cd8bb75ce37269c418b9f5632ac18d86d888e5031e50"} Jul 10 19:44:45.044813 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.044755 2579 scope.go:117] "RemoveContainer" containerID="0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d" Jul 10 19:44:45.054379 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.054363 2579 scope.go:117] "RemoveContainer" containerID="0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d" Jul 10 19:44:45.054616 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:44:45.054597 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d\": container with ID starting with 0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d not found: ID does not exist" containerID="0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d" Jul 10 19:44:45.054684 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.054624 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d"} err="failed to get container status \"0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d\": rpc error: code = NotFound desc = could not find container \"0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d\": container with ID starting with 0ce083eecad95a912e718f53a37f5501cce8489a7497532653db6b19d633de4d not found: ID does not exist" Jul 10 19:44:45.071822 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.071788 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c44fac5-a83f-41ce-a047-a74063e1fffb" path="/var/lib/kubelet/pods/3c44fac5-a83f-41ce-a047-a74063e1fffb/volumes" Jul 10 19:44:45.072081 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.072068 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr"] Jul 10 19:44:45.072262 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.072245 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/auth-test-model-fc4b18-kserve-5b6bd4df7f-trbnr"] Jul 10 19:44:45.134317 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.134290 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:45.134464 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.134321 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:45.134464 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:45.134337 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6mwc\" (UniqueName: \"kubernetes.io/projected/e8a2f9f2-3438-4ada-ae2b-78c15f08a377-kube-api-access-x6mwc\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:44:47.070568 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:44:47.070525 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a2f9f2-3438-4ada-ae2b-78c15f08a377" path="/var/lib/kubelet/pods/e8a2f9f2-3438-4ada-ae2b-78c15f08a377/volumes" Jul 10 19:45:02.999567 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:02.999486 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz"] Jul 10 19:45:03.001881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:02.999908 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c44fac5-a83f-41ce-a047-a74063e1fffb" containerName="main" Jul 10 19:45:03.001881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:02.999919 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c44fac5-a83f-41ce-a047-a74063e1fffb" containerName="main" Jul 10 19:45:03.001881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:02.999941 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a2f9f2-3438-4ada-ae2b-78c15f08a377" containerName="main" Jul 10 19:45:03.001881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:02.999947 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a2f9f2-3438-4ada-ae2b-78c15f08a377" containerName="main" Jul 10 19:45:03.001881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.000013 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c44fac5-a83f-41ce-a047-a74063e1fffb" containerName="main" Jul 10 19:45:03.001881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.000024 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a2f9f2-3438-4ada-ae2b-78c15f08a377" containerName="main" Jul 10 19:45:03.002853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.002839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.006232 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.006209 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"sub-test-model-fc4b18-kserve-self-signed-certs\"" Jul 10 19:45:03.007272 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.007258 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"kube-root-ca.crt\"" Jul 10 19:45:03.007338 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.007269 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"openshift-service-ca.crt\"" Jul 10 19:45:03.007525 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.007512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"default-dockercfg-988nc\"" Jul 10 19:45:03.040896 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.040869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz"] Jul 10 19:45:03.070307 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.070278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:03.195147 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.195109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb8xv\" (UniqueName: \"kubernetes.io/projected/3c89c469-c5dc-46b0-b4b5-22372c906eb8-kube-api-access-nb8xv\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.195311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.195184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tmp-dir\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.195311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.195228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-dshm\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.195311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.195265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-model-cache\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.195311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.195294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tls-certs\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.195483 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.195389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-home\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296355 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tmp-dir\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296355 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-dshm\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296355 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-model-cache\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296642 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tls-certs\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296642 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-home\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296642 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb8xv\" (UniqueName: \"kubernetes.io/projected/3c89c469-c5dc-46b0-b4b5-22372c906eb8-kube-api-access-nb8xv\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296839 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tmp-dir\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.296839 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.296750 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-model-cache\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.297071 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.297038 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-home\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.298617 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.298588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-dshm\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.298877 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.298860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tls-certs\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.306311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.306285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb8xv\" (UniqueName: \"kubernetes.io/projected/3c89c469-c5dc-46b0-b4b5-22372c906eb8-kube-api-access-nb8xv\") pod \"sub-test-model-fc4b18-kserve-74598c79ff-wq6sz\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.312174 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.312152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:03.466198 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:03.466173 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz"] Jul 10 19:45:03.468132 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:45:03.468104 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c89c469_c5dc_46b0_b4b5_22372c906eb8.slice/crio-5b8efbb7f174a43a517b5abe23146a525122920abb980be31656b5ce1832ef96 WatchSource:0}: Error finding container 5b8efbb7f174a43a517b5abe23146a525122920abb980be31656b5ce1832ef96: Status 404 returned error can't find the container with id 5b8efbb7f174a43a517b5abe23146a525122920abb980be31656b5ce1832ef96 Jul 10 19:45:04.138700 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:04.138667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" event={"ID":"3c89c469-c5dc-46b0-b4b5-22372c906eb8","Type":"ContainerStarted","Data":"3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6"} Jul 10 19:45:04.138700 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:04.138707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" event={"ID":"3c89c469-c5dc-46b0-b4b5-22372c906eb8","Type":"ContainerStarted","Data":"5b8efbb7f174a43a517b5abe23146a525122920abb980be31656b5ce1832ef96"} Jul 10 19:45:04.167226 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:04.167178 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" podStartSLOduration=2.167163137 podStartE2EDuration="2.167163137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:45:04.165856281 +0000 UTC m=+2244.204729299" watchObservedRunningTime="2026-07-10 19:45:04.167163137 +0000 UTC m=+2244.206036153" Jul 10 19:45:13.312366 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:13.312334 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:13.312736 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:13.312382 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:13.324595 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:13.324575 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:14.196723 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:14.196697 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:45:45.840809 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:45.840774 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g"] Jul 10 19:45:45.882544 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:45.882518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:46.921783 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:46.921741 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz"] Jul 10 19:45:46.951298 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:45:46.951266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:46:00.198258 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:00.198225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:46:00.860744 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:00.860712 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" podUID="729c7370-4e8e-4a5c-bf25-2581fa218bfd" containerName="main" containerID="cri-o://896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50" gracePeriod=45 Jul 10 19:46:01.105796 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.105774 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:46:01.203613 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.203532 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tls-certs\") pod \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " Jul 10 19:46:01.203613 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.203576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-model-cache\") pod \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " Jul 10 19:46:01.204108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.203620 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m5ms\" (UniqueName: \"kubernetes.io/projected/729c7370-4e8e-4a5c-bf25-2581fa218bfd-kube-api-access-9m5ms\") pod \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " Jul 10 19:46:01.204108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.203692 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-dshm\") pod \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " Jul 10 19:46:01.204108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.203760 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tmp-dir\") pod \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " Jul 10 19:46:01.204108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.203814 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-home\") pod \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\" (UID: \"729c7370-4e8e-4a5c-bf25-2581fa218bfd\") " Jul 10 19:46:01.204108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.203902 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "729c7370-4e8e-4a5c-bf25-2581fa218bfd" (UID: "729c7370-4e8e-4a5c-bf25-2581fa218bfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:01.204108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.204019 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "729c7370-4e8e-4a5c-bf25-2581fa218bfd" (UID: "729c7370-4e8e-4a5c-bf25-2581fa218bfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:01.204108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.204095 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-home" (OuterVolumeSpecName: "home") pod "729c7370-4e8e-4a5c-bf25-2581fa218bfd" (UID: "729c7370-4e8e-4a5c-bf25-2581fa218bfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:01.204408 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.204232 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:01.204408 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.204252 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:01.204408 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.204291 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:01.205831 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.205808 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "729c7370-4e8e-4a5c-bf25-2581fa218bfd" (UID: "729c7370-4e8e-4a5c-bf25-2581fa218bfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:46:01.205901 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.205821 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-dshm" (OuterVolumeSpecName: "dshm") pod "729c7370-4e8e-4a5c-bf25-2581fa218bfd" (UID: "729c7370-4e8e-4a5c-bf25-2581fa218bfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:01.205901 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.205841 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729c7370-4e8e-4a5c-bf25-2581fa218bfd-kube-api-access-9m5ms" (OuterVolumeSpecName: "kube-api-access-9m5ms") pod "729c7370-4e8e-4a5c-bf25-2581fa218bfd" (UID: "729c7370-4e8e-4a5c-bf25-2581fa218bfd"). InnerVolumeSpecName "kube-api-access-9m5ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:46:01.305300 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.305274 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/729c7370-4e8e-4a5c-bf25-2581fa218bfd-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:01.305300 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.305297 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/729c7370-4e8e-4a5c-bf25-2581fa218bfd-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:01.305449 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.305308 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m5ms\" (UniqueName: \"kubernetes.io/projected/729c7370-4e8e-4a5c-bf25-2581fa218bfd-kube-api-access-9m5ms\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:01.430115 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.430083 2579 generic.go:356] "Generic (PLEG): container finished" podID="729c7370-4e8e-4a5c-bf25-2581fa218bfd" containerID="896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50" exitCode=0 Jul 10 19:46:01.430279 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.430129 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" Jul 10 19:46:01.430279 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.430167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" event={"ID":"729c7370-4e8e-4a5c-bf25-2581fa218bfd","Type":"ContainerDied","Data":"896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50"} Jul 10 19:46:01.430279 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.430212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g" event={"ID":"729c7370-4e8e-4a5c-bf25-2581fa218bfd","Type":"ContainerDied","Data":"55b6d32c32fa89c6f53eb693646bdb26ca5d816aacdf0e234ff79e60d3a2977a"} Jul 10 19:46:01.430279 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.430229 2579 scope.go:117] "RemoveContainer" containerID="896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50" Jul 10 19:46:01.439237 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.439222 2579 scope.go:117] "RemoveContainer" containerID="896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50" Jul 10 19:46:01.439492 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:46:01.439474 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50\": container with ID starting with 896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50 not found: ID does not exist" containerID="896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50" Jul 10 19:46:01.439537 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.439502 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50"} err="failed to get container status \"896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50\": rpc error: code = NotFound desc = could not find container \"896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50\": container with ID starting with 896f3b8b7fe3de328e9afe266c220a9deb53cd374d57b6961c9034681a0e0e50 not found: ID does not exist" Jul 10 19:46:01.468840 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.468787 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g"] Jul 10 19:46:01.476637 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.476619 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f25aa8/sub-test-model-f25aa8-kserve-699d4bff69-47g7g"] Jul 10 19:46:01.943289 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:01.943257 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" podUID="3c89c469-c5dc-46b0-b4b5-22372c906eb8" containerName="main" containerID="cri-o://3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6" gracePeriod=45 Jul 10 19:46:02.190388 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.190366 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:46:02.213325 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213261 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tmp-dir\") pod \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " Jul 10 19:46:02.213714 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213339 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-dshm\") pod \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " Jul 10 19:46:02.213714 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213369 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tls-certs\") pod \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " Jul 10 19:46:02.213714 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213421 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-home\") pod \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " Jul 10 19:46:02.213714 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb8xv\" (UniqueName: \"kubernetes.io/projected/3c89c469-c5dc-46b0-b4b5-22372c906eb8-kube-api-access-nb8xv\") pod \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " Jul 10 19:46:02.213714 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213478 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c89c469-c5dc-46b0-b4b5-22372c906eb8" (UID: "3c89c469-c5dc-46b0-b4b5-22372c906eb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:02.213714 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213489 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-model-cache\") pod \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\" (UID: \"3c89c469-c5dc-46b0-b4b5-22372c906eb8\") " Jul 10 19:46:02.214062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-home" (OuterVolumeSpecName: "home") pod "3c89c469-c5dc-46b0-b4b5-22372c906eb8" (UID: "3c89c469-c5dc-46b0-b4b5-22372c906eb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:02.214062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213799 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c89c469-c5dc-46b0-b4b5-22372c906eb8" (UID: "3c89c469-c5dc-46b0-b4b5-22372c906eb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:02.214062 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.213804 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:02.215483 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.215462 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c89c469-c5dc-46b0-b4b5-22372c906eb8-kube-api-access-nb8xv" (OuterVolumeSpecName: "kube-api-access-nb8xv") pod "3c89c469-c5dc-46b0-b4b5-22372c906eb8" (UID: "3c89c469-c5dc-46b0-b4b5-22372c906eb8"). InnerVolumeSpecName "kube-api-access-nb8xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:46:02.215594 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.215505 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-dshm" (OuterVolumeSpecName: "dshm") pod "3c89c469-c5dc-46b0-b4b5-22372c906eb8" (UID: "3c89c469-c5dc-46b0-b4b5-22372c906eb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:46:02.215594 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.215505 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c89c469-c5dc-46b0-b4b5-22372c906eb8" (UID: "3c89c469-c5dc-46b0-b4b5-22372c906eb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:46:02.314269 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.314238 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:02.314269 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.314266 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb8xv\" (UniqueName: \"kubernetes.io/projected/3c89c469-c5dc-46b0-b4b5-22372c906eb8-kube-api-access-nb8xv\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:02.314269 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.314277 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:02.314492 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.314287 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c89c469-c5dc-46b0-b4b5-22372c906eb8-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:02.314492 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.314295 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c89c469-c5dc-46b0-b4b5-22372c906eb8-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:46:02.436012 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.435982 2579 generic.go:356] "Generic (PLEG): container finished" podID="3c89c469-c5dc-46b0-b4b5-22372c906eb8" containerID="3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6" exitCode=0 Jul 10 19:46:02.436174 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.436022 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" Jul 10 19:46:02.436174 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.436059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" event={"ID":"3c89c469-c5dc-46b0-b4b5-22372c906eb8","Type":"ContainerDied","Data":"3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6"} Jul 10 19:46:02.436174 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.436095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz" event={"ID":"3c89c469-c5dc-46b0-b4b5-22372c906eb8","Type":"ContainerDied","Data":"5b8efbb7f174a43a517b5abe23146a525122920abb980be31656b5ce1832ef96"} Jul 10 19:46:02.436174 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.436117 2579 scope.go:117] "RemoveContainer" containerID="3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6" Jul 10 19:46:02.445617 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.445601 2579 scope.go:117] "RemoveContainer" containerID="3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6" Jul 10 19:46:02.445904 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:46:02.445880 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6\": container with ID starting with 3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6 not found: ID does not exist" containerID="3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6" Jul 10 19:46:02.445981 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.445914 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6"} err="failed to get container status \"3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6\": rpc error: code = NotFound desc = could not find container \"3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6\": container with ID starting with 3bbe6cb89c88ab14006b1deeeffb696c7bd399043e04f1706f6910230b7874f6 not found: ID does not exist" Jul 10 19:46:02.464807 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.464753 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz"] Jul 10 19:46:02.469830 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:02.469812 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/sub-test-model-fc4b18-kserve-74598c79ff-wq6sz"] Jul 10 19:46:03.071761 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:03.071729 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c89c469-c5dc-46b0-b4b5-22372c906eb8" path="/var/lib/kubelet/pods/3c89c469-c5dc-46b0-b4b5-22372c906eb8/volumes" Jul 10 19:46:03.072124 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:03.072111 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729c7370-4e8e-4a5c-bf25-2581fa218bfd" path="/var/lib/kubelet/pods/729c7370-4e8e-4a5c-bf25-2581fa218bfd/volumes" Jul 10 19:46:31.791748 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.791716 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c"] Jul 10 19:46:31.792205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.792107 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729c7370-4e8e-4a5c-bf25-2581fa218bfd" containerName="main" Jul 10 19:46:31.792205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.792118 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="729c7370-4e8e-4a5c-bf25-2581fa218bfd" containerName="main" Jul 10 19:46:31.792205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.792152 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c89c469-c5dc-46b0-b4b5-22372c906eb8" containerName="main" Jul 10 19:46:31.792205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.792158 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c89c469-c5dc-46b0-b4b5-22372c906eb8" containerName="main" Jul 10 19:46:31.792337 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.792220 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c89c469-c5dc-46b0-b4b5-22372c906eb8" containerName="main" Jul 10 19:46:31.792337 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.792228 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="729c7370-4e8e-4a5c-bf25-2581fa218bfd" containerName="main" Jul 10 19:46:31.795695 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.795676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:31.799083 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.799063 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"default-dockercfg-988nc\"" Jul 10 19:46:31.799539 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.799521 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"openshift-service-ca.crt\"" Jul 10 19:46:31.800389 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.800371 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"kube-root-ca.crt\"" Jul 10 19:46:31.800481 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.800435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fc4b18\"/\"rate-test-model-fc4b18-kserve-self-signed-certs\"" Jul 10 19:46:31.806071 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.806051 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c"] Jul 10 19:46:31.832303 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.832279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:46:31.966319 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.966294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2242f95-522f-4d50-820c-e525858e89e9-tls-certs\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:31.966414 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.966335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-model-cache\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:31.966414 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.966359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759gc\" (UniqueName: \"kubernetes.io/projected/a2242f95-522f-4d50-820c-e525858e89e9-kube-api-access-759gc\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:31.966503 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.966428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-dshm\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:31.966503 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.966495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-home\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:31.966580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:31.966526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-tmp-dir\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.067439 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-dshm\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.067580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-home\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.067580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-tmp-dir\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.067580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2242f95-522f-4d50-820c-e525858e89e9-tls-certs\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.067783 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-model-cache\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.067783 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759gc\" (UniqueName: \"kubernetes.io/projected/a2242f95-522f-4d50-820c-e525858e89e9-kube-api-access-759gc\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.067916 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-home\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.068042 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-tmp-dir\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.068042 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.067967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-model-cache\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.069582 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.069558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-dshm\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.069838 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.069822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2242f95-522f-4d50-820c-e525858e89e9-tls-certs\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.077814 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.077795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759gc\" (UniqueName: \"kubernetes.io/projected/a2242f95-522f-4d50-820c-e525858e89e9-kube-api-access-759gc\") pod \"rate-test-model-fc4b18-kserve-7d4846d88c-85b9c\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.104795 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.104742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:32.238014 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.237990 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c"] Jul 10 19:46:32.240044 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:46:32.240013 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2242f95_522f_4d50_820c_e525858e89e9.slice/crio-27dfaffa271f4deb788df4bcb1d976f4174b16a5e61f46795576d4a7981a4614 WatchSource:0}: Error finding container 27dfaffa271f4deb788df4bcb1d976f4174b16a5e61f46795576d4a7981a4614: Status 404 returned error can't find the container with id 27dfaffa271f4deb788df4bcb1d976f4174b16a5e61f46795576d4a7981a4614 Jul 10 19:46:32.582495 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.582457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" event={"ID":"a2242f95-522f-4d50-820c-e525858e89e9","Type":"ContainerStarted","Data":"113ede87226129667e7fee7089af7f25f619d4dd3904f4c35bdeda2ec8146438"} Jul 10 19:46:32.582671 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.582502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" event={"ID":"a2242f95-522f-4d50-820c-e525858e89e9","Type":"ContainerStarted","Data":"27dfaffa271f4deb788df4bcb1d976f4174b16a5e61f46795576d4a7981a4614"} Jul 10 19:46:32.605392 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:32.605343 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" podStartSLOduration=1.605328515 podStartE2EDuration="1.605328515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:46:32.603387179 +0000 UTC m=+2332.642260193" watchObservedRunningTime="2026-07-10 19:46:32.605328515 +0000 UTC m=+2332.644201582" Jul 10 19:46:42.105001 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:42.104970 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:42.105398 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:42.105009 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:42.117451 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:42.117427 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:42.637544 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:42.637519 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:46:48.866956 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:46:48.866924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:09.606881 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:09.606840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:09.916119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:09.916029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:10.019060 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:10.019016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:10.592323 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:10.592286 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c"] Jul 10 19:47:10.617314 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:10.617288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:25.612117 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:25.612078 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" podUID="a2242f95-522f-4d50-820c-e525858e89e9" containerName="main" containerID="cri-o://113ede87226129667e7fee7089af7f25f619d4dd3904f4c35bdeda2ec8146438" gracePeriod=45 Jul 10 19:47:25.841839 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:25.841808 2579 generic.go:356] "Generic (PLEG): container finished" podID="a2242f95-522f-4d50-820c-e525858e89e9" containerID="113ede87226129667e7fee7089af7f25f619d4dd3904f4c35bdeda2ec8146438" exitCode=0 Jul 10 19:47:25.841977 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:25.841860 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" event={"ID":"a2242f95-522f-4d50-820c-e525858e89e9","Type":"ContainerDied","Data":"113ede87226129667e7fee7089af7f25f619d4dd3904f4c35bdeda2ec8146438"} Jul 10 19:47:25.865564 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:25.865507 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:47:26.016558 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.016529 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759gc\" (UniqueName: \"kubernetes.io/projected/a2242f95-522f-4d50-820c-e525858e89e9-kube-api-access-759gc\") pod \"a2242f95-522f-4d50-820c-e525858e89e9\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " Jul 10 19:47:26.016728 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.016621 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2242f95-522f-4d50-820c-e525858e89e9-tls-certs\") pod \"a2242f95-522f-4d50-820c-e525858e89e9\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " Jul 10 19:47:26.016728 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.016686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-dshm\") pod \"a2242f95-522f-4d50-820c-e525858e89e9\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " Jul 10 19:47:26.016856 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.016740 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-home\") pod \"a2242f95-522f-4d50-820c-e525858e89e9\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " Jul 10 19:47:26.016856 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.016765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-tmp-dir\") pod \"a2242f95-522f-4d50-820c-e525858e89e9\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " Jul 10 19:47:26.016856 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.016788 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-model-cache\") pod \"a2242f95-522f-4d50-820c-e525858e89e9\" (UID: \"a2242f95-522f-4d50-820c-e525858e89e9\") " Jul 10 19:47:26.017065 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.017021 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-home" (OuterVolumeSpecName: "home") pod "a2242f95-522f-4d50-820c-e525858e89e9" (UID: "a2242f95-522f-4d50-820c-e525858e89e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:47:26.017065 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.017029 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2242f95-522f-4d50-820c-e525858e89e9" (UID: "a2242f95-522f-4d50-820c-e525858e89e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:47:26.017188 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.017101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2242f95-522f-4d50-820c-e525858e89e9" (UID: "a2242f95-522f-4d50-820c-e525858e89e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:47:26.017188 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.017152 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-home\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:26.017188 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.017170 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-tmp-dir\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:26.018807 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.018785 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-dshm" (OuterVolumeSpecName: "dshm") pod "a2242f95-522f-4d50-820c-e525858e89e9" (UID: "a2242f95-522f-4d50-820c-e525858e89e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:47:26.018882 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.018803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2242f95-522f-4d50-820c-e525858e89e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2242f95-522f-4d50-820c-e525858e89e9" (UID: "a2242f95-522f-4d50-820c-e525858e89e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:47:26.018882 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.018814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2242f95-522f-4d50-820c-e525858e89e9-kube-api-access-759gc" (OuterVolumeSpecName: "kube-api-access-759gc") pod "a2242f95-522f-4d50-820c-e525858e89e9" (UID: "a2242f95-522f-4d50-820c-e525858e89e9"). InnerVolumeSpecName "kube-api-access-759gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:47:26.117708 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.117621 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2242f95-522f-4d50-820c-e525858e89e9-tls-certs\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:26.117708 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.117643 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-dshm\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:26.117708 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.117675 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2242f95-522f-4d50-820c-e525858e89e9-model-cache\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:26.117708 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.117687 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759gc\" (UniqueName: \"kubernetes.io/projected/a2242f95-522f-4d50-820c-e525858e89e9-kube-api-access-759gc\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:26.848073 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.848044 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" Jul 10 19:47:26.848476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.848076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c" event={"ID":"a2242f95-522f-4d50-820c-e525858e89e9","Type":"ContainerDied","Data":"27dfaffa271f4deb788df4bcb1d976f4174b16a5e61f46795576d4a7981a4614"} Jul 10 19:47:26.848476 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.848126 2579 scope.go:117] "RemoveContainer" containerID="113ede87226129667e7fee7089af7f25f619d4dd3904f4c35bdeda2ec8146438" Jul 10 19:47:26.870564 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.870540 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c"] Jul 10 19:47:26.880853 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:26.880835 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fc4b18/rate-test-model-fc4b18-kserve-7d4846d88c-85b9c"] Jul 10 19:47:27.072428 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:27.072400 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2242f95-522f-4d50-820c-e525858e89e9" path="/var/lib/kubelet/pods/a2242f95-522f-4d50-820c-e525858e89e9/volumes" Jul 10 19:47:30.176665 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:30.176629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:30.670833 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:30.670805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:30.769298 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:30.769273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:30.867782 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:30.867755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:30.999031 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:30.998949 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg"] Jul 10 19:47:30.999214 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:30.999173 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" podUID="b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" containerName="maas-api" containerID="cri-o://1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa" gracePeriod=30 Jul 10 19:47:31.033872 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.033845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:31.449042 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.449018 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:47:31.568919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.568853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-openshift-service-ca\") pod \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " Jul 10 19:47:31.568919 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.568898 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-maas-api-tls\") pod \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " Jul 10 19:47:31.569079 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.568975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcvxj\" (UniqueName: \"kubernetes.io/projected/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-kube-api-access-hcvxj\") pod \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\" (UID: \"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7\") " Jul 10 19:47:31.569172 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.569153 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" (UID: "b6bb8f87-96ec-4ed4-b58a-c8b6663819b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:47:31.569266 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.569254 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:31.570958 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.570936 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" (UID: "b6bb8f87-96ec-4ed4-b58a-c8b6663819b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:47:31.571057 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.571032 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-kube-api-access-hcvxj" (OuterVolumeSpecName: "kube-api-access-hcvxj") pod "b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" (UID: "b6bb8f87-96ec-4ed4-b58a-c8b6663819b7"). InnerVolumeSpecName "kube-api-access-hcvxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:47:31.670871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.670836 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvxj\" (UniqueName: \"kubernetes.io/projected/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-kube-api-access-hcvxj\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:31.670871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.670871 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:31.873577 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.873550 2579 generic.go:356] "Generic (PLEG): container finished" podID="b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" containerID="1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa" exitCode=0 Jul 10 19:47:31.873734 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.873596 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" Jul 10 19:47:31.873734 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.873635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" event={"ID":"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7","Type":"ContainerDied","Data":"1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa"} Jul 10 19:47:31.873734 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.873702 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg" event={"ID":"b6bb8f87-96ec-4ed4-b58a-c8b6663819b7","Type":"ContainerDied","Data":"c457d16713d8d1a903aa6fea065dd3acd3c537c8b3660a4a5ed8d6830f8549d6"} Jul 10 19:47:31.873734 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.873725 2579 scope.go:117] "RemoveContainer" containerID="1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa" Jul 10 19:47:31.883171 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.883153 2579 scope.go:117] "RemoveContainer" containerID="1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa" Jul 10 19:47:31.883419 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:47:31.883400 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa\": container with ID starting with 1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa not found: ID does not exist" containerID="1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa" Jul 10 19:47:31.883464 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.883428 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa"} err="failed to get container status \"1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa\": rpc error: code = NotFound desc = could not find container \"1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa\": container with ID starting with 1878c03c17ae5600cc02fc144f7ea1e8acf5334b431b83818a8c39c9c7e874fa not found: ID does not exist" Jul 10 19:47:31.895860 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.895840 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg"] Jul 10 19:47:31.900317 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:31.900299 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f25aa8-565b7f9fbf-n9jlg"] Jul 10 19:47:33.071281 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:33.071242 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" path="/var/lib/kubelet/pods/b6bb8f87-96ec-4ed4-b58a-c8b6663819b7/volumes" Jul 10 19:47:36.957465 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:36.957425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:37.059175 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:37.059145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:37.328982 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:37.328899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:37.518712 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:37.518675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:37.731923 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:37.731895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:38.134746 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.134720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:38.236333 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.236306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:38.336850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.336825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:38.512034 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.511946 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf"] Jul 10 19:47:38.512194 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.512176 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" podUID="8d3af1ea-1bb4-4742-adf6-c6550980c8d8" containerName="maas-api" containerID="cri-o://3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667" gracePeriod=30 Jul 10 19:47:38.542846 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.542812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:38.768450 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.768396 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:47:38.826061 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.826027 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-openshift-service-ca\") pod \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " Jul 10 19:47:38.826061 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.826070 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls\") pod \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " Jul 10 19:47:38.826280 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.826093 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx6lb\" (UniqueName: \"kubernetes.io/projected/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-kube-api-access-vx6lb\") pod \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\" (UID: \"8d3af1ea-1bb4-4742-adf6-c6550980c8d8\") " Jul 10 19:47:38.826431 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.826409 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d3af1ea-1bb4-4742-adf6-c6550980c8d8" (UID: "8d3af1ea-1bb4-4742-adf6-c6550980c8d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:47:38.828039 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.828020 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d3af1ea-1bb4-4742-adf6-c6550980c8d8" (UID: "8d3af1ea-1bb4-4742-adf6-c6550980c8d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:47:38.828117 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.828087 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-kube-api-access-vx6lb" (OuterVolumeSpecName: "kube-api-access-vx6lb") pod "8d3af1ea-1bb4-4742-adf6-c6550980c8d8" (UID: "8d3af1ea-1bb4-4742-adf6-c6550980c8d8"). InnerVolumeSpecName "kube-api-access-vx6lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:47:38.912480 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.912446 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d3af1ea-1bb4-4742-adf6-c6550980c8d8" containerID="3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667" exitCode=0 Jul 10 19:47:38.912624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.912493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" Jul 10 19:47:38.912624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.912537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" event={"ID":"8d3af1ea-1bb4-4742-adf6-c6550980c8d8","Type":"ContainerDied","Data":"3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667"} Jul 10 19:47:38.912624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.912594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf" event={"ID":"8d3af1ea-1bb4-4742-adf6-c6550980c8d8","Type":"ContainerDied","Data":"ca18a8f060db35e74edb1534a7b411483da0ca14c383bcd9b3415bc818b4b206"} Jul 10 19:47:38.912624 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.912619 2579 scope.go:117] "RemoveContainer" containerID="3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667" Jul 10 19:47:38.922040 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.922025 2579 scope.go:117] "RemoveContainer" containerID="3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667" Jul 10 19:47:38.922308 ip-10-0-132-221 kubenswrapper[2579]: E0710 19:47:38.922290 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667\": container with ID starting with 3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667 not found: ID does not exist" containerID="3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667" Jul 10 19:47:38.922372 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.922315 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667"} err="failed to get container status \"3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667\": rpc error: code = NotFound desc = could not find container \"3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667\": container with ID starting with 3c77237a735ca696a2d770131950486e0940d20054ecf22a124d0ef6ca4d1667 not found: ID does not exist" Jul 10 19:47:38.927728 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.927711 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-maas-api-tls\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:38.927795 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.927730 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx6lb\" (UniqueName: \"kubernetes.io/projected/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-kube-api-access-vx6lb\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:38.927795 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.927740 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3af1ea-1bb4-4742-adf6-c6550980c8d8-openshift-service-ca\") on node \"ip-10-0-132-221.ec2.internal\" DevicePath \"\"" Jul 10 19:47:38.936740 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.936720 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf"] Jul 10 19:47:38.941823 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:38.941800 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fc4b18-749ccd857-q5vgf"] Jul 10 19:47:39.072728 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:39.072674 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d3af1ea-1bb4-4742-adf6-c6550980c8d8" path="/var/lib/kubelet/pods/8d3af1ea-1bb4-4742-adf6-c6550980c8d8/volumes" Jul 10 19:47:41.072338 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:41.072308 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:47:41.080620 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:41.080601 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tm9q6_1a153daf-5790-42c6-9c33-98f40f104f91/ovn-acl-logging/0.log" Jul 10 19:47:44.343426 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:44.343392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:44.446205 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:44.446178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:44.643278 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:44.643199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:44.886238 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:44.886169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:51.077478 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:51.077451 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d686dc7f-7j97s_f399cf7a-6480-4d4a-8771-985942ec039d/copy-manifests/0.log" Jul 10 19:47:51.084403 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:51.084382 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d686dc7f-7j97s_f399cf7a-6480-4d4a-8771-985942ec039d/manager/0.log" Jul 10 19:47:51.742747 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:51.742713 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b4cfccc4d-kb4m5_82ba7900-8012-4776-b19c-ae1fc1ec52dc/maas-api/0.log" Jul 10 19:47:51.880341 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:51.880319 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-fj2ql_d90297b8-8226-4aef-86f9-d3f8262c1e8c/postgres/0.log" Jul 10 19:47:53.105907 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:53.105882 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-9gxd9_293eb0dd-0cdf-432d-95f4-8997f3aeaeec/kuadrant-console-plugin/0.log" Jul 10 19:47:54.148359 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:54.148327 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-967644b46-ckv89_6eb62fa8-498a-4445-bf23-b5389bd56434/kube-auth-proxy/0.log" Jul 10 19:47:54.632619 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:54.632589 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gmpsm_b6a19137-9eac-4215-a5e2-d8c38972c332/payload-processing/0.log" Jul 10 19:47:55.260423 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:55.260398 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2dlfm_0e7a28d7-af9f-4d9e-9b78-d3e542d021e3/main/0.log" Jul 10 19:47:55.498311 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:55.498282 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblnpfr_48346b08-fc9c-4e8a-9c54-e1f106b0e585/main/0.log" Jul 10 19:47:59.784191 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784158 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kffvc/must-gather-cv45s"] Jul 10 19:47:59.784560 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784547 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" containerName="maas-api" Jul 10 19:47:59.784560 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784561 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" containerName="maas-api" Jul 10 19:47:59.784639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784571 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d3af1ea-1bb4-4742-adf6-c6550980c8d8" containerName="maas-api" Jul 10 19:47:59.784639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784577 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d3af1ea-1bb4-4742-adf6-c6550980c8d8" containerName="maas-api" Jul 10 19:47:59.784639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784592 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2242f95-522f-4d50-820c-e525858e89e9" containerName="main" Jul 10 19:47:59.784639 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784599 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2242f95-522f-4d50-820c-e525858e89e9" containerName="main" Jul 10 19:47:59.784792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784692 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bb8f87-96ec-4ed4-b58a-c8b6663819b7" containerName="maas-api" Jul 10 19:47:59.784792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784703 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d3af1ea-1bb4-4742-adf6-c6550980c8d8" containerName="maas-api" Jul 10 19:47:59.784792 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.784713 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2242f95-522f-4d50-820c-e525858e89e9" containerName="main" Jul 10 19:47:59.787932 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.787912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:47:59.791973 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.791953 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kffvc\"/\"openshift-service-ca.crt\"" Jul 10 19:47:59.794502 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.794482 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kffvc\"/\"kube-root-ca.crt\"" Jul 10 19:47:59.833815 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.833791 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kffvc/must-gather-cv45s"] Jul 10 19:47:59.889112 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.889084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:47:59.908274 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.908253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jflmw\" (UniqueName: \"kubernetes.io/projected/dd9181f9-7905-4843-8a9a-e0b0860557f5-kube-api-access-jflmw\") pod \"must-gather-cv45s\" (UID: \"dd9181f9-7905-4843-8a9a-e0b0860557f5\") " pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:47:59.908330 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:47:59.908313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd9181f9-7905-4843-8a9a-e0b0860557f5-must-gather-output\") pod \"must-gather-cv45s\" (UID: \"dd9181f9-7905-4843-8a9a-e0b0860557f5\") " pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:48:00.009333 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.009308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jflmw\" (UniqueName: \"kubernetes.io/projected/dd9181f9-7905-4843-8a9a-e0b0860557f5-kube-api-access-jflmw\") pod \"must-gather-cv45s\" (UID: \"dd9181f9-7905-4843-8a9a-e0b0860557f5\") " pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:48:00.009426 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.009368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd9181f9-7905-4843-8a9a-e0b0860557f5-must-gather-output\") pod \"must-gather-cv45s\" (UID: \"dd9181f9-7905-4843-8a9a-e0b0860557f5\") " pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:48:00.009702 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.009686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dd9181f9-7905-4843-8a9a-e0b0860557f5-must-gather-output\") pod \"must-gather-cv45s\" (UID: \"dd9181f9-7905-4843-8a9a-e0b0860557f5\") " pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:48:00.020904 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.020879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jflmw\" (UniqueName: \"kubernetes.io/projected/dd9181f9-7905-4843-8a9a-e0b0860557f5-kube-api-access-jflmw\") pod \"must-gather-cv45s\" (UID: \"dd9181f9-7905-4843-8a9a-e0b0860557f5\") " pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:48:00.062329 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.062263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" limit=1024 Jul 10 19:48:00.096871 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.096848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kffvc/must-gather-cv45s" Jul 10 19:48:00.235793 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.235773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kffvc/must-gather-cv45s"] Jul 10 19:48:00.238095 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:48:00.238073 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd9181f9_7905_4843_8a9a_e0b0860557f5.slice/crio-c297ee5e7721f57b3ccd79cfe3a439d7b13574bcc44354eafc25dc942e6d8992 WatchSource:0}: Error finding container c297ee5e7721f57b3ccd79cfe3a439d7b13574bcc44354eafc25dc942e6d8992: Status 404 returned error can't find the container with id c297ee5e7721f57b3ccd79cfe3a439d7b13574bcc44354eafc25dc942e6d8992 Jul 10 19:48:00.239872 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:00.239840 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:48:01.018451 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:01.018415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kffvc/must-gather-cv45s" event={"ID":"dd9181f9-7905-4843-8a9a-e0b0860557f5","Type":"ContainerStarted","Data":"c297ee5e7721f57b3ccd79cfe3a439d7b13574bcc44354eafc25dc942e6d8992"} Jul 10 19:48:02.025679 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:02.025631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kffvc/must-gather-cv45s" event={"ID":"dd9181f9-7905-4843-8a9a-e0b0860557f5","Type":"ContainerStarted","Data":"e860b05b97755bfaf23cf4dd93f5d142a7169a733d9860f1516b90bd8729efd3"} Jul 10 19:48:02.026084 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:02.025688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kffvc/must-gather-cv45s" event={"ID":"dd9181f9-7905-4843-8a9a-e0b0860557f5","Type":"ContainerStarted","Data":"38c9a0c86941a620230194f6c507e6f658c85011bcd1da00895688cd8feebe7c"} Jul 10 19:48:02.044720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:02.044648 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kffvc/must-gather-cv45s" podStartSLOduration=2.088411483 podStartE2EDuration="3.044632732s" totalImagesPullingTime="956.221249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:48:02.043375718 +0000 UTC m=+2422.082248795" watchObservedRunningTime="2026-07-10 19:48:02.044632732 +0000 UTC m=+2422.083505749" Jul 10 19:48:02.902171 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:02.902146 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ssw5f_d1acace1-4b9b-402d-a6ab-f8d8f3a401cc/global-pull-secret-syncer/0.log" Jul 10 19:48:02.958544 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:02.958520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cj2bx_86f3ee5e-5fae-4ce8-b74f-c0d34d1b8608/konnectivity-agent/0.log" Jul 10 19:48:03.066560 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:03.066509 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-221.ec2.internal_c868ebd70961df44ee76abf8462d8081/haproxy/0.log" Jul 10 19:48:07.656018 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:07.655990 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-9gxd9_293eb0dd-0cdf-432d-95f4-8997f3aeaeec/kuadrant-console-plugin/0.log" Jul 10 19:48:09.458444 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.458410 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16e4ba2d-9c2b-4d65-a06f-9b067539826a/alertmanager/0.log" Jul 10 19:48:09.494114 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.494089 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16e4ba2d-9c2b-4d65-a06f-9b067539826a/config-reloader/0.log" Jul 10 19:48:09.520887 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.520864 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16e4ba2d-9c2b-4d65-a06f-9b067539826a/kube-rbac-proxy-web/0.log" Jul 10 19:48:09.549574 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.549524 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16e4ba2d-9c2b-4d65-a06f-9b067539826a/kube-rbac-proxy/0.log" Jul 10 19:48:09.587035 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.586955 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16e4ba2d-9c2b-4d65-a06f-9b067539826a/kube-rbac-proxy-metric/0.log" Jul 10 19:48:09.614536 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.614497 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16e4ba2d-9c2b-4d65-a06f-9b067539826a/prom-label-proxy/0.log" Jul 10 19:48:09.639169 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.639113 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16e4ba2d-9c2b-4d65-a06f-9b067539826a/init-config-reloader/0.log" Jul 10 19:48:09.815625 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.815549 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5cff96cdbf-rvvvp_767bee55-2625-4e10-a3b5-867390c1ee06/metrics-server/0.log" Jul 10 19:48:09.967041 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.966996 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6p2dl_b58e1945-6536-410c-bb40-90e374750ee1/node-exporter/0.log" Jul 10 19:48:09.991537 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:09.991512 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6p2dl_b58e1945-6536-410c-bb40-90e374750ee1/kube-rbac-proxy/0.log" Jul 10 19:48:10.022903 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.022870 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6p2dl_b58e1945-6536-410c-bb40-90e374750ee1/init-textfile/0.log" Jul 10 19:48:10.265580 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.265476 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_677af96a-5af1-468b-a579-a91cd84b46ca/prometheus/0.log" Jul 10 19:48:10.294203 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.294181 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_677af96a-5af1-468b-a579-a91cd84b46ca/config-reloader/0.log" Jul 10 19:48:10.330005 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.329976 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_677af96a-5af1-468b-a579-a91cd84b46ca/thanos-sidecar/0.log" Jul 10 19:48:10.358009 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.357892 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_677af96a-5af1-468b-a579-a91cd84b46ca/kube-rbac-proxy-web/0.log" Jul 10 19:48:10.382527 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.382486 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_677af96a-5af1-468b-a579-a91cd84b46ca/kube-rbac-proxy/0.log" Jul 10 19:48:10.410332 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.410307 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_677af96a-5af1-468b-a579-a91cd84b46ca/kube-rbac-proxy-thanos/0.log" Jul 10 19:48:10.435696 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.435673 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_677af96a-5af1-468b-a579-a91cd84b46ca/init-config-reloader/0.log" Jul 10 19:48:10.471211 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.471185 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-tnkss_8fbf6a90-cbaf-41a7-abc3-de3f54620f0a/prometheus-operator/0.log" Jul 10 19:48:10.496344 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.496316 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-tnkss_8fbf6a90-cbaf-41a7-abc3-de3f54620f0a/kube-rbac-proxy/0.log" Jul 10 19:48:10.525858 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.525807 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-dpf2v_c90b0659-9974-4e23-92d8-351dccb06ce0/prometheus-operator-admission-webhook/0.log" Jul 10 19:48:10.560638 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.560595 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f6775cdb4-8wpn6_ddaba4d3-8e03-4800-b845-2bc0b535aa97/telemeter-client/0.log" Jul 10 19:48:10.588167 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.588120 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f6775cdb4-8wpn6_ddaba4d3-8e03-4800-b845-2bc0b535aa97/reload/0.log" Jul 10 19:48:10.614000 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.613876 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f6775cdb4-8wpn6_ddaba4d3-8e03-4800-b845-2bc0b535aa97/kube-rbac-proxy/0.log" Jul 10 19:48:10.657507 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.657464 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76c5dfb559-fg88n_0ab4184a-cba3-448c-b2cc-422618f2a863/thanos-query/0.log" Jul 10 19:48:10.680353 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.680331 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76c5dfb559-fg88n_0ab4184a-cba3-448c-b2cc-422618f2a863/kube-rbac-proxy-web/0.log" Jul 10 19:48:10.705645 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.705619 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76c5dfb559-fg88n_0ab4184a-cba3-448c-b2cc-422618f2a863/kube-rbac-proxy/0.log" Jul 10 19:48:10.732887 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.732853 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76c5dfb559-fg88n_0ab4184a-cba3-448c-b2cc-422618f2a863/prom-label-proxy/0.log" Jul 10 19:48:10.758720 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.758678 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76c5dfb559-fg88n_0ab4184a-cba3-448c-b2cc-422618f2a863/kube-rbac-proxy-rules/0.log" Jul 10 19:48:10.782474 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:10.782442 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76c5dfb559-fg88n_0ab4184a-cba3-448c-b2cc-422618f2a863/kube-rbac-proxy-metrics/0.log" Jul 10 19:48:12.053973 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.053935 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-85xd9_e68db36c-a90d-481c-9ce5-720c1610e7c4/networking-console-plugin/0.log" Jul 10 19:48:12.332685 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.332586 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw"] Jul 10 19:48:12.338358 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.338325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.344442 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.344411 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kffvc\"/\"default-dockercfg-c6tm2\"" Jul 10 19:48:12.349800 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.348901 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw"] Jul 10 19:48:12.449108 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.449069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-sys\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.449305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.449163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-proc\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.449305 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.449252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-lib-modules\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.449431 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.449381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-podres\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.449510 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.449483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftjzj\" (UniqueName: \"kubernetes.io/projected/45cdce08-65f5-4a0d-9331-086cb326973f-kube-api-access-ftjzj\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551113 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-podres\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftjzj\" (UniqueName: \"kubernetes.io/projected/45cdce08-65f5-4a0d-9331-086cb326973f-kube-api-access-ftjzj\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-sys\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551301 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-podres\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-sys\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-proc\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551479 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-lib-modules\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551640 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-lib-modules\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.551640 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.551557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45cdce08-65f5-4a0d-9331-086cb326973f-proc\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.561410 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.561376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftjzj\" (UniqueName: \"kubernetes.io/projected/45cdce08-65f5-4a0d-9331-086cb326973f-kube-api-access-ftjzj\") pod \"perf-node-gather-daemonset-2zcrw\" (UID: \"45cdce08-65f5-4a0d-9331-086cb326973f\") " pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.656746 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.656709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:12.837888 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:12.836119 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw"] Jul 10 19:48:12.838706 ip-10-0-132-221 kubenswrapper[2579]: W0710 19:48:12.838257 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod45cdce08_65f5_4a0d_9331_086cb326973f.slice/crio-93b098802047e6c10877838f3caad9a0a6cd2e13dda5aa75bb2cb6814f0c5515 WatchSource:0}: Error finding container 93b098802047e6c10877838f3caad9a0a6cd2e13dda5aa75bb2cb6814f0c5515: Status 404 returned error can't find the container with id 93b098802047e6c10877838f3caad9a0a6cd2e13dda5aa75bb2cb6814f0c5515 Jul 10 19:48:13.101264 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:13.101223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" event={"ID":"45cdce08-65f5-4a0d-9331-086cb326973f","Type":"ContainerStarted","Data":"38ab0f7d3e8c63eb15785b79c06c1c3b1dccee9ce6734e2abe3e16e0eaf6adad"} Jul 10 19:48:13.101696 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:13.101274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" event={"ID":"45cdce08-65f5-4a0d-9331-086cb326973f","Type":"ContainerStarted","Data":"93b098802047e6c10877838f3caad9a0a6cd2e13dda5aa75bb2cb6814f0c5515"} Jul 10 19:48:13.101696 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:13.101365 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:13.126154 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:13.126095 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" podStartSLOduration=1.126074479 podStartE2EDuration="1.126074479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:48:13.123907952 +0000 UTC m=+2433.162780971" watchObservedRunningTime="2026-07-10 19:48:13.126074479 +0000 UTC m=+2433.164947497" Jul 10 19:48:13.361561 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:13.361535 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fd6dbb547-np9wr_1087aae5-7079-4efc-9684-9f77d24d3a5a/console/0.log" Jul 10 19:48:13.397253 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:13.397228 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-fh6rn_a7b4d22f-9d66-4edc-b47d-e60671ac5979/download-server/0.log" Jul 10 19:48:15.152850 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:15.152821 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8pfjr_fa44e407-2dce-40f8-9886-a9d779b16f6c/dns/0.log" Jul 10 19:48:15.178295 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:15.178259 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8pfjr_fa44e407-2dce-40f8-9886-a9d779b16f6c/kube-rbac-proxy/0.log" Jul 10 19:48:15.348326 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:15.348295 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sxcqz_54573428-5115-497b-9056-2ebd4783c0a5/dns-node-resolver/0.log" Jul 10 19:48:15.901058 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:15.901024 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75fb498fbd-hgkfj_1d7665b8-2fb9-4e5f-93f9-bb1a7bd166a3/registry/0.log" Jul 10 19:48:15.924667 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:15.924620 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfwrf_c5a5c819-e84a-4d88-8c7a-8be2fa966e33/node-ca/0.log" Jul 10 19:48:17.027140 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:17.027105 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-967644b46-ckv89_6eb62fa8-498a-4445-bf23-b5389bd56434/kube-auth-proxy/0.log" Jul 10 19:48:17.159536 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:17.159506 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gmpsm_b6a19137-9eac-4215-a5e2-d8c38972c332/payload-processing/0.log" Jul 10 19:48:17.897924 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:17.897899 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n85lh_9c49025d-6cfa-4ef3-ba99-40a5ccc707ab/serve-healthcheck-canary/0.log" Jul 10 19:48:18.449593 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:18.449536 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6v88z_c9364374-b9ef-440a-af41-696eb14832c8/kube-rbac-proxy/0.log" Jul 10 19:48:18.475865 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:18.475836 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6v88z_c9364374-b9ef-440a-af41-696eb14832c8/exporter/0.log" Jul 10 19:48:18.502848 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:18.502826 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6v88z_c9364374-b9ef-440a-af41-696eb14832c8/extractor/0.log" Jul 10 19:48:19.120176 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:19.120148 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kffvc/perf-node-gather-daemonset-2zcrw" Jul 10 19:48:21.428618 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:21.428595 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d686dc7f-7j97s_f399cf7a-6480-4d4a-8771-985942ec039d/manager/0.log" Jul 10 19:48:21.454450 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:21.454329 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d686dc7f-7j97s_f399cf7a-6480-4d4a-8771-985942ec039d/copy-manifests/0.log" Jul 10 19:48:22.926433 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:22.926397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d6cfdfcb9-dd747_681431d1-a903-49f2-bf27-80c9a909982d/manager/0.log" Jul 10 19:48:23.033419 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:23.033387 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-967644b46-ckv89_6eb62fa8-498a-4445-bf23-b5389bd56434/kube-auth-proxy/0.log" Jul 10 19:48:23.126634 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:23.126593 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gmpsm_b6a19137-9eac-4215-a5e2-d8c38972c332/payload-processing/0.log" Jul 10 19:48:24.489414 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:24.489383 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-967644b46-ckv89_6eb62fa8-498a-4445-bf23-b5389bd56434/kube-auth-proxy/0.log" Jul 10 19:48:24.577371 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:24.577334 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-gmpsm_b6a19137-9eac-4215-a5e2-d8c38972c332/payload-processing/0.log" Jul 10 19:48:29.436706 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.436632 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4tsnc_6a34835a-6cb8-4fb1-a5c8-445574ab2fa1/kube-multus/0.log" Jul 10 19:48:29.466117 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.466091 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4r25b_1309dd30-f208-401e-a5e0-b33b8a018b90/kube-multus-additional-cni-plugins/0.log" Jul 10 19:48:29.493119 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.493094 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4r25b_1309dd30-f208-401e-a5e0-b33b8a018b90/egress-router-binary-copy/0.log" Jul 10 19:48:29.518924 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.518903 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4r25b_1309dd30-f208-401e-a5e0-b33b8a018b90/cni-plugins/0.log" Jul 10 19:48:29.543892 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.543868 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4r25b_1309dd30-f208-401e-a5e0-b33b8a018b90/bond-cni-plugin/0.log" Jul 10 19:48:29.575920 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.575892 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4r25b_1309dd30-f208-401e-a5e0-b33b8a018b90/routeoverride-cni/0.log" Jul 10 19:48:29.603983 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.603949 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4r25b_1309dd30-f208-401e-a5e0-b33b8a018b90/whereabouts-cni-bincopy/0.log" Jul 10 19:48:29.630484 ip-10-0-132-221 kubenswrapper[2579]: I0710 19:48:29.630454 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4r25b_1309dd30-f208-401e-a5e0-b33b8a018b90/whereabouts-cni/0.log"