Jul 09 14:14:37.982784 ip-10-0-142-0 systemd[1]: Starting Kubernetes Kubelet... Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.441617 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447121 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447136 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447141 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447146 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447151 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447155 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447159 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:14:38.482139 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447163 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447168 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447172 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447176 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447180 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447184 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447188 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447192 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447196 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447200 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447204 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447207 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447211 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447215 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447220 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447224 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447227 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447231 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447236 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447241 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:14:38.483189 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447244 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447248 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447252 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447256 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447260 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447264 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447269 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447272 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447277 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447281 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447286 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447292 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447297 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447300 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447305 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447309 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447313 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447317 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447321 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:14:38.486649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447325 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447329 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447333 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447337 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447341 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447345 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447349 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447353 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447356 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447360 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447364 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447368 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447372 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447376 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447380 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447384 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447387 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447394 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447399 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447404 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:14:38.487226 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447408 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447411 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447416 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447420 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447424 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447429 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447433 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447437 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447441 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447446 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447451 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447455 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447458 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447462 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447465 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447469 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447473 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447477 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447480 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:14:38.487793 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447484 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447488 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447492 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447495 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447500 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447504 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447508 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447511 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447515 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447519 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447535 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447540 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447544 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.447547 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448502 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448510 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448514 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448519 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448523 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448542 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:14:38.488369 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448545 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448550 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448554 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448558 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448563 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448567 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448573 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448578 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448582 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448587 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448590 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448594 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448598 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448602 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448606 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448612 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448618 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448623 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448627 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:14:39.857228 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448632 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:14:38.532050 ip-10-0-142-0 systemd[1]: Started Kubernetes Kubelet. Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448636 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448640 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448643 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448647 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448651 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448655 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448658 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448662 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448666 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448669 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448673 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448677 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448681 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448684 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448688 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448692 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448696 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448701 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448704 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:14:39.858278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448707 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448711 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448715 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448718 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448722 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448726 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448730 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448734 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448738 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448742 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448746 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448750 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448755 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448758 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448762 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448765 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448769 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448773 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448776 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:14:39.859278 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448780 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448783 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448787 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448790 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448794 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448797 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448802 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448806 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448810 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448814 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448818 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448822 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448825 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448830 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448833 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448837 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448841 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448847 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448851 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448855 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:14:39.860113 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448858 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448862 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448865 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448869 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448872 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448876 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448880 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448883 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448887 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448891 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448895 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448899 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448903 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448906 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.448910 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449426 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449433 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449437 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449441 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449445 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:14:39.860927 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449449 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449455 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449459 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449465 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449471 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449475 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449479 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449483 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449487 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449493 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449496 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449500 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449504 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449508 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449511 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449515 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449519 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449523 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449542 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449546 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:14:39.861754 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449550 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449554 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449558 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449562 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449566 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449570 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449573 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449577 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449581 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449584 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449588 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449592 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449597 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449604 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449607 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449611 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449616 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449622 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449628 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449632 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:14:39.862891 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449636 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449642 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449646 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449650 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449654 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449657 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449661 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449665 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449669 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449672 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449676 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449680 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449683 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449687 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449691 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449695 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449698 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449702 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449706 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449709 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:14:39.863871 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449713 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449717 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449720 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449724 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449728 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449737 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449741 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449745 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449748 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449752 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449756 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449760 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449765 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449771 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449775 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449779 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449783 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449786 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449790 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:14:39.864705 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449794 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449797 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449801 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449805 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449808 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449812 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449816 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449820 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449824 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449828 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449832 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449836 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449839 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449843 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.449847 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.449939 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.449961 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.449973 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.449980 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.449986 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.449991 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 14:14:39.865809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.449997 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450003 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450009 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450014 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450019 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450026 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450030 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450035 2556 flags.go:64] FLAG: --cgroup-root="" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450039 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450043 2556 flags.go:64] FLAG: --client-ca-file="" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450048 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450052 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450058 2556 flags.go:64] FLAG: --cluster-domain="" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450062 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450066 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450071 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450075 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450080 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450085 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450089 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450093 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450098 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450102 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450106 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 14:14:39.866759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450111 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450115 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450121 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450125 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450129 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450136 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450140 2556 flags.go:64] FLAG: --enable-server="true" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450144 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450150 2556 flags.go:64] FLAG: --event-burst="100" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450154 2556 flags.go:64] FLAG: --event-qps="50" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450158 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450163 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450168 2556 flags.go:64] FLAG: --eviction-hard="" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450175 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450179 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450183 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450187 2556 flags.go:64] FLAG: --eviction-soft="" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450191 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450196 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450200 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450204 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450209 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450213 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450217 2556 flags.go:64] FLAG: --feature-gates="" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450222 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 14:14:39.867835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450226 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450231 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450235 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450240 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450244 2556 flags.go:64] FLAG: --help="false" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450248 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-142-0.ec2.internal" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450252 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450257 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450262 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450267 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450272 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450277 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450283 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450287 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450291 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450295 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450299 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450304 2556 flags.go:64] FLAG: --kube-reserved="" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450308 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450313 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450321 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450326 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450330 2556 flags.go:64] FLAG: --lock-file="" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450334 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 14:14:39.869133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450338 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450342 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450350 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450354 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450358 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450362 2556 flags.go:64] FLAG: --logging-format="text" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450366 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450371 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450375 2556 flags.go:64] FLAG: --manifest-url="" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450379 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450385 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450389 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450395 2556 flags.go:64] FLAG: --max-pods="110" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450399 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450404 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450408 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450412 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450416 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450421 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450425 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450436 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450440 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450445 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 14:14:39.870185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450449 2556 flags.go:64] FLAG: --pod-cidr="" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450453 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450461 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450465 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450469 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450475 2556 flags.go:64] FLAG: --port="10250" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450481 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450485 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02b141c749b7e9599" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450490 2556 flags.go:64] FLAG: --qos-reserved="" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450494 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450498 2556 flags.go:64] FLAG: --register-node="true" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450502 2556 flags.go:64] FLAG: --register-with-taints="" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450508 2556 flags.go:64] FLAG: --registry-burst="10" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450512 2556 flags.go:64] FLAG: --registry-qps="5" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450516 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450519 2556 flags.go:64] FLAG: --reserved-memory="" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450539 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450544 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450549 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450553 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450558 2556 flags.go:64] FLAG: --runonce="false" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450562 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450567 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450571 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450575 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 14:14:39.870821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450579 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450584 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450588 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450593 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450598 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450602 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450606 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450610 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450615 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450619 2556 flags.go:64] FLAG: --system-cgroups="" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450623 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450631 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450636 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450642 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450648 2556 flags.go:64] FLAG: --tls-min-version="" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450652 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450656 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450660 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450664 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450668 2556 flags.go:64] FLAG: --v="2" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450674 2556 flags.go:64] FLAG: --version="false" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450679 2556 flags.go:64] FLAG: --vmodule="" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450684 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.450689 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450869 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:14:39.871680 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450875 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450879 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450883 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450887 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450891 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450895 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450898 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450902 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450909 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450914 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450918 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450924 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450928 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450932 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450935 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450939 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450943 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450947 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450950 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450954 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:14:39.873288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450959 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450964 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450968 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450972 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450976 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450980 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450983 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450987 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450991 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450994 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.450998 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451001 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451005 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451009 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451013 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451017 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451020 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451024 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451028 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:14:39.874350 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451032 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451035 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451040 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451043 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451048 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451052 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451056 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451060 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451064 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451067 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451071 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451075 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451079 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451084 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451088 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451093 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451096 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451100 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451104 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451108 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:14:40.017484 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451111 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451115 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451119 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451123 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451129 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451133 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451138 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451142 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451145 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451149 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451153 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451157 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451163 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451168 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451172 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451176 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451182 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451186 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451190 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:14:40.033329 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451194 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451198 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451202 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451206 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451210 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451213 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451219 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451222 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451226 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451230 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451234 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451238 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451242 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451245 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451249 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451253 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451256 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451260 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451264 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:14:40.033999 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.451268 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.451935 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.464472 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.464488 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.464501 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.464507 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464584 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464590 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464593 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464597 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464600 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464603 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464605 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464608 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464611 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:14:40.034606 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464613 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464616 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464619 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464621 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464624 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464627 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464629 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464633 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464636 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464640 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464644 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464647 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464650 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464654 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464657 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464659 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464662 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464665 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464667 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:14:40.035245 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464670 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464672 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464675 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464678 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464681 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464683 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464686 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464688 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464691 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464693 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464696 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464698 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464701 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464704 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464706 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464709 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464711 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464713 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464716 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464719 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:14:40.035862 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464721 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464736 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464740 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464743 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464746 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464748 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464752 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464755 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464757 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464760 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464763 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464765 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464768 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464771 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464773 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464776 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464778 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464781 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464783 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464786 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:14:40.036685 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464789 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464791 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464794 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464796 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464799 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464801 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464803 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464806 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464808 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464811 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464813 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464816 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464818 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464820 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464829 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464831 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464834 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464836 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464839 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464842 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:14:40.037304 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464845 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464847 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464850 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464853 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464856 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464858 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464861 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464867 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464870 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464873 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.464876 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.464881 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465016 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465021 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:14:40.037921 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465024 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465027 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465030 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465033 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465036 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465039 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465042 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465045 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465047 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465050 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465052 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465055 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465058 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465067 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465070 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465072 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465075 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465077 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465080 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:14:40.038341 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465082 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465085 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465087 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465090 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465094 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465097 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465101 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465103 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465106 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465109 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465111 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465114 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465116 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465119 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465121 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465124 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465126 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465129 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465131 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465134 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:14:40.038939 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465136 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465139 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465141 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465144 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465146 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465149 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465151 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465159 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465162 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465164 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465167 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465169 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465172 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465174 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465177 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465179 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465183 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465185 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465190 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465194 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:14:40.039543 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465197 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465200 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465203 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465206 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465209 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465212 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465215 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465217 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465220 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465223 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465226 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465229 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465232 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465234 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465236 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465239 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465241 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465244 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465246 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465249 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:14:40.040088 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465257 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465260 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465262 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465265 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465267 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465271 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465274 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465277 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465281 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465284 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465289 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465292 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465294 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465297 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465299 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465302 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465304 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:14:40.040675 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:38.465307 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.465312 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.465413 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.467594 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.469001 2556 server.go:1019] "Starting client certificate rotation" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.469096 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.469896 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.471653 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.476799 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.491733 2556 log.go:25] "Validated CRI v1 runtime API" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.494178 2556 log.go:25] "Validated CRI v1 image API" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.494941 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 14:14:40.041204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.504929 2556 fs.go:135] Filesystem UUIDs: map[524cff4f-a363-483c-a89f-d625fbc3265e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a9d5ff2f-3a5e-43f9-9f79-2fdd060146c6:/dev/nvme0n1p3] Jul 09 14:14:40.041621 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.504945 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.509426 2556 manager.go:216] Machine: {Timestamp:2026-07-09 14:14:38.508891962 +0000 UTC m=+0.404301148 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096183 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec271c9feace09e53dd0ae359e563f13 SystemUUID:ec271c9f-eace-09e5-3dd0-ae359e563f13 BootID:a795e0cf-b4f3-4461-b93e-2b6178ce7815 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:6a:9c:81:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:6a:9c:81:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:96:8f:c6:86:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.509550 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.509620 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.511394 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.511421 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-0.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.512173 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.512187 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 14:14:40.041694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.512197 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.512215 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.513190 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.513450 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.515941 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516074 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516104 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516116 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516136 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516148 2556 kubelet.go:398] "Adding apiserver pod source" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516159 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516506 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.516538 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.521590 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.522942 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.522978 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524451 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524469 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524491 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524501 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524514 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524522 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524548 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524557 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524563 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524568 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524583 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.524588 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.525628 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.525635 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.528837 2556 server.go:1268] "Started kubelet" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.528914 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.528902 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.530885 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.531177 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-0.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:38.531388 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:38.531638 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.531911 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.540156 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlznn" Jul 09 14:14:40.041998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.541352 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.544933 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.544961 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:38.545168 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.545817 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.545837 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.545848 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.545983 2556 factory.go:55] Registering systemd factory Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.545998 2556 factory.go:223] Registration of the systemd container factory successfully Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546011 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546032 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:38.546046 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546185 2556 factory.go:153] Registering CRI-O factory Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546197 2556 factory.go:223] Registration of the crio container factory successfully Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546241 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546274 2556 factory.go:103] Registering Raw factory Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546284 2556 manager.go:1196] Started watching for new ooms in manager Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:38.545464 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-0.ec2.internal.18c0a4a46bcb9e26 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-0.ec2.internal,UID:ip-10-0-142-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-0.ec2.internal,},FirstTimestamp:2026-07-09 14:14:38.52881463 +0000 UTC m=+0.424223816,LastTimestamp:2026-07-09 14:14:38.52881463 +0000 UTC m=+0.424223816,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-0.ec2.internal,}" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.546654 2556 manager.go:349] Starting recovery of all containers Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:38.548085 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlznn" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.848499 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:39.851324 2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-0.ec2.internal\" not found" node="ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.853606 2556 manager.go:354] Recovery completed Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.860141 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.862165 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.862190 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.862200 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.862689 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.862701 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.862719 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.862690 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.863392 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.863431 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.863449 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.863913 2556 policy_none.go:49] "None policy: Start" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.863929 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 14:14:40.043021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.863944 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.865029 2556 policy_none.go:47] "Start" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.907017 2556 manager.go:337] "Starting Device Plugin manager" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:39.907051 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.907061 2556 server.go:87] "Starting device plugin registration server" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.907309 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.907322 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.907524 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.907647 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:39.907660 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:39.908920 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:39.908970 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.007613 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.008580 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.008605 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.008615 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.008639 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.017568 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.017589 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-0.ec2.internal\": node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.044158 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.038518 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.067945 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.069118 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.069151 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.069169 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.069201 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.071958 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.140045 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.170182 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal"] Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.170243 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.171297 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.171924 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.172133 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.174297 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.174588 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.174620 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.175120 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.175139 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.175150 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.175389 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.175403 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.175415 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.176593 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.176619 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.177446 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.177468 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:14:40.180629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.177480 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:14:40.305201 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.214956 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-0.ec2.internal\" not found" node="ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.305201 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.219098 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-0.ec2.internal\" not found" node="ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.305201 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.240508 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.440758 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.341061 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.440758 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.341130 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b4097f5baac0499a3289c3b380cb8c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal\" (UID: \"1b4097f5baac0499a3289c3b380cb8c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.440758 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.341151 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b4097f5baac0499a3289c3b380cb8c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal\" (UID: \"1b4097f5baac0499a3289c3b380cb8c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.440758 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.341167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23cb67f50f5c5e7503dadf6a1637939a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-0.ec2.internal\" (UID: \"23cb67f50f5c5e7503dadf6a1637939a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.442602 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.441491 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.442602 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.441540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b4097f5baac0499a3289c3b380cb8c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal\" (UID: \"1b4097f5baac0499a3289c3b380cb8c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.442602 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.441558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b4097f5baac0499a3289c3b380cb8c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal\" (UID: \"1b4097f5baac0499a3289c3b380cb8c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.442602 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.441566 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b4097f5baac0499a3289c3b380cb8c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal\" (UID: \"1b4097f5baac0499a3289c3b380cb8c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.442602 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.441582 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23cb67f50f5c5e7503dadf6a1637939a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-0.ec2.internal\" (UID: \"23cb67f50f5c5e7503dadf6a1637939a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.442602 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.441605 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23cb67f50f5c5e7503dadf6a1637939a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-0.ec2.internal\" (UID: \"23cb67f50f5c5e7503dadf6a1637939a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.442602 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.441608 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b4097f5baac0499a3289c3b380cb8c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal\" (UID: \"1b4097f5baac0499a3289c3b380cb8c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.561857 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.517687 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.561857 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.522638 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.561857 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.542045 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.680133 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.642447 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.742875 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-0.ec2.internal\" not found" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.759020 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.759162 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.759189 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.775832 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.807839 2556 apiserver.go:52] "Watching apiserver" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.818474 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.818862 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-tmpjn","openshift-cluster-node-tuning-operator/tuned-c9bb6","openshift-image-registry/node-ca-cj6rf","openshift-multus/multus-7qrm8","openshift-multus/multus-additional-cni-plugins-hjpww","openshift-ovn-kubernetes/ovnkube-node-hhjnx","kube-system/konnectivity-agent-b9g5l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs","openshift-dns/node-resolver-ch6ct","openshift-multus/network-metrics-daemon-s7f2d","openshift-network-diagnostics/network-check-target-b5ssg","openshift-network-operator/iptables-alerter-hjh24"] Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.820497 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.820564 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.822766 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.825009 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.825111 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.825172 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhhjg\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.825310 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.827013 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.827627 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.827647 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zzl52\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.827663 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.827648 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.828305 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.829199 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.829558 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.830308 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.830406 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.830521 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 14:14:40.840590 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.830594 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tfwcm\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.830720 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.833294 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.834782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.836137 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.836231 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.836258 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.836313 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.837933 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.837992 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.839294 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.840621 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.840635 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.840748 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.840835 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.840999 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841014 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841030 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841041 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841036 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841169 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n5s5t\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841278 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841335 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841366 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841368 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841573 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fm4md\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841639 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841679 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841782 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jxfsv\"" Jul 09 14:14:40.842543 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841850 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.841902 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.842006 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6ttcw\"" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.842132 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zl7cb\"" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.842133 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5krs4\"" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.842192 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.842986 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe2b570d-f7c6-426d-badd-136411182962-cni-binary-copy\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.842953 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 14:09:39 +0000 UTC" deadline="2028-01-16 21:51:42.796302319 +0000 UTC" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843032 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-k8s-cni-cncf-io\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843035 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13351h37m1.953272605s" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-ovn\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-node-log\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843101 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-kubelet-config\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843121 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843143 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysctl-conf\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843163 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-lib-modules\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.843638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843200 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843235 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843264 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-log-socket\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843285 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-cnibin\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843308 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-cni-bin\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843332 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843357 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-env-overrides\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843394 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-var-lib-kubelet\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843459 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cnibin\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843496 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjg6\" (UniqueName: \"kubernetes.io/projected/fe2b570d-f7c6-426d-badd-136411182962-kube-api-access-nvjg6\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843521 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-kubelet\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843542 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-device-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843561 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-run\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843600 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-host\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.844659 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843621 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-conf-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843661 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-etc-kubernetes\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.843683 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/911301db-2aff-4ec6-ab8a-f613769de8e0-agent-certs\") pod \"konnectivity-agent-b9g5l\" (UID: \"911301db-2aff-4ec6-ab8a-f613769de8e0\") " pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844057 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-etc-selinux\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844102 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysconfig\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844135 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-cni-bin\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844164 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-sys-fs\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844188 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x624s\" (UniqueName: \"kubernetes.io/projected/35fedfd8-5445-48d3-b505-f22acb89d199-kube-api-access-x624s\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844218 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-dbus\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844366 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mbj\" (UniqueName: \"kubernetes.io/projected/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-kube-api-access-f6mbj\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844410 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-socket-dir-parent\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844483 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-cni-multus\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844524 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-kubelet\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844558 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.845449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844583 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-kubernetes\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-hostroot\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844702 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-var-lib-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844729 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovnkube-config\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/911301db-2aff-4ec6-ab8a-f613769de8e0-konnectivity-ca\") pod \"konnectivity-agent-b9g5l\" (UID: \"911301db-2aff-4ec6-ab8a-f613769de8e0\") " pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844795 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-kubelet-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844842 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-systemd\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844889 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.844928 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe2b570d-f7c6-426d-badd-136411182962-multus-daemon-config\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-systemd-units\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-os-release\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-host\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-serviceca\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.846403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845184 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-run-netns\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-system-cni-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845241 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-etc-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845271 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7h8t\" (UniqueName: \"kubernetes.io/projected/a892d681-79c6-4216-b544-5f5db8314362-kube-api-access-p7h8t\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-system-cni-dir\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845480 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysctl-d\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-multus-certs\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845533 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whx9r\" (UniqueName: \"kubernetes.io/projected/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-kube-api-access-whx9r\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-socket-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845591 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfl4\" (UniqueName: \"kubernetes.io/projected/6a19f14e-5a01-44e1-9933-dca850de7aea-kube-api-access-jbfl4\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845623 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-modprobe-d\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845655 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fhv\" (UniqueName: \"kubernetes.io/projected/d8b08c62-97d7-4fea-a7ba-54773582ceaf-kube-api-access-m5fhv\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845804 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-cni-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845833 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-netns\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.847459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845865 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845896 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-registration-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-slash\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845959 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-systemd\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.845986 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-sys\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846015 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a892d681-79c6-4216-b544-5f5db8314362-etc-tuned\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846042 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a892d681-79c6-4216-b544-5f5db8314362-tmp\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846074 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846393 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85fhs\" (UniqueName: \"kubernetes.io/projected/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-kube-api-access-85fhs\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846450 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-os-release\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846497 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-cni-netd\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846524 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovn-node-metrics-cert\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846548 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovnkube-script-lib\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846578 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a19f14e-5a01-44e1-9933-dca850de7aea-hosts-file\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.849894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.846610 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19f14e-5a01-44e1-9933-dca850de7aea-tmp-dir\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.855828 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.857359 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal"] Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.857514 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.857601 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.870785 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal"] Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.870982 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.903396 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qlnct" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.917262 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qlnct" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.937416 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.946943 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-cni-bin\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.946965 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-sys-fs\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.946982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x624s\" (UniqueName: \"kubernetes.io/projected/35fedfd8-5445-48d3-b505-f22acb89d199-kube-api-access-x624s\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.946998 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-dbus\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947021 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mbj\" (UniqueName: \"kubernetes.io/projected/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-kube-api-access-f6mbj\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947068 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-sys-fs\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947121 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-dbus\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947132 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-socket-dir-parent\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-socket-dir-parent\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.987095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947176 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-cni-multus\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947068 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-cni-bin\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947200 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-kubelet\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947234 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-cni-multus\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-kubernetes\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947255 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-kubelet\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947265 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947293 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-hostroot\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947304 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-kubernetes\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947318 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1314eb6f-01a6-472d-8a2b-85128dbe3545-host-slash\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947340 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-hostroot\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947341 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-var-lib-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947363 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovnkube-config\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947366 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-var-lib-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947384 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/911301db-2aff-4ec6-ab8a-f613769de8e0-konnectivity-ca\") pod \"konnectivity-agent-b9g5l\" (UID: \"911301db-2aff-4ec6-ab8a-f613769de8e0\") " pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947405 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-kubelet-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947442 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-systemd\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947465 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947487 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe2b570d-f7c6-426d-badd-136411182962-multus-daemon-config\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947508 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtr5q\" (UniqueName: \"kubernetes.io/projected/1314eb6f-01a6-472d-8a2b-85128dbe3545-kube-api-access-wtr5q\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947530 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-systemd-units\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-os-release\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947570 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-host\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-serviceca\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947609 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-run-netns\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947629 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-system-cni-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947653 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-etc-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.988570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947683 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7h8t\" (UniqueName: \"kubernetes.io/projected/a892d681-79c6-4216-b544-5f5db8314362-kube-api-access-p7h8t\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947706 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-system-cni-dir\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysctl-d\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947751 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-multus-certs\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947773 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whx9r\" (UniqueName: \"kubernetes.io/projected/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-kube-api-access-whx9r\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947794 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-socket-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfl4\" (UniqueName: \"kubernetes.io/projected/6a19f14e-5a01-44e1-9933-dca850de7aea-kube-api-access-jbfl4\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-modprobe-d\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947867 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fhv\" (UniqueName: \"kubernetes.io/projected/d8b08c62-97d7-4fea-a7ba-54773582ceaf-kube-api-access-m5fhv\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-cni-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947909 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-netns\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1314eb6f-01a6-472d-8a2b-85128dbe3545-iptables-alerter-script\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-registration-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.989230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.947994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-slash\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948015 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-systemd\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948036 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-sys\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a892d681-79c6-4216-b544-5f5db8314362-etc-tuned\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948079 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a892d681-79c6-4216-b544-5f5db8314362-tmp\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85fhs\" (UniqueName: \"kubernetes.io/projected/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-kube-api-access-85fhs\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948151 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-os-release\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-cni-netd\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948195 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovn-node-metrics-cert\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948216 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovnkube-script-lib\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948243 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a19f14e-5a01-44e1-9933-dca850de7aea-hosts-file\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19f14e-5a01-44e1-9933-dca850de7aea-tmp-dir\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948295 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948316 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe2b570d-f7c6-426d-badd-136411182962-cni-binary-copy\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.989747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948338 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-k8s-cni-cncf-io\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948357 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-ovn\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948377 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-node-log\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948400 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-kubelet-config\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948460 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysctl-conf\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-lib-modules\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948504 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948526 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948547 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-log-socket\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-cnibin\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948597 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-cni-bin\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948620 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-env-overrides\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.990379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948666 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-var-lib-kubelet\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948689 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cnibin\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948711 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjg6\" (UniqueName: \"kubernetes.io/projected/fe2b570d-f7c6-426d-badd-136411182962-kube-api-access-nvjg6\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948733 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-kubelet\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948755 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-device-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948777 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-run\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-host\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-conf-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948854 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-etc-kubernetes\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948876 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/911301db-2aff-4ec6-ab8a-f613769de8e0-agent-certs\") pod \"konnectivity-agent-b9g5l\" (UID: \"911301db-2aff-4ec6-ab8a-f613769de8e0\") " pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948897 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-etc-selinux\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.948918 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysconfig\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysconfig\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949059 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-kubelet-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-systemd\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.990842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949368 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949406 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-cni-netd\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949589 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-log-socket\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949606 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-ovn\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949644 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-kubelet\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949663 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-kubelet-config\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949704 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-cnibin\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.949745 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949749 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-var-lib-cni-bin\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949793 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949832 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-device-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.949844 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret podName:e988dcf9-c3bd-4bc1-95d0-d5217096ec97 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:41.449823551 +0000 UTC m=+2.054338247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret") pod "global-pull-secret-syncer-tmpjn" (UID: "e988dcf9-c3bd-4bc1-95d0-d5217096ec97") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949886 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-run\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe2b570d-f7c6-426d-badd-136411182962-multus-daemon-config\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949932 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-host\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.991657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949988 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-etc-kubernetes\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.949993 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-systemd-units\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19f14e-5a01-44e1-9933-dca850de7aea-tmp-dir\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950043 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-host\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950063 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-node-log\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950087 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-systemd\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950120 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-k8s-cni-cncf-io\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950164 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-os-release\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950209 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-conf-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950209 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-env-overrides\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950232 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovnkube-config\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950253 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cnibin\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950287 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe2b570d-f7c6-426d-badd-136411182962-cni-binary-copy\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.950298 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.950356 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs podName:03bbda19-f3b2-4d1d-8aaa-1c1647b55efa nodeName:}" failed. No retries permitted until 2026-07-09 14:14:41.450343216 +0000 UTC m=+2.054857894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs") pod "network-metrics-daemon-s7f2d" (UID: "03bbda19-f3b2-4d1d-8aaa-1c1647b55efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:40.992130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950373 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-etc-selinux\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950391 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-run-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysctl-conf\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-netns\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950447 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-multus-cni-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950480 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-registration-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950509 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-slash\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950568 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-sysctl-d\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950596 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-host-run-netns\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950625 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-system-cni-dir\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950621 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovnkube-script-lib\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950659 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-etc-openvswitch\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950706 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-lib-modules\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950755 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a19f14e-5a01-44e1-9933-dca850de7aea-hosts-file\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950830 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b08c62-97d7-4fea-a7ba-54773582ceaf-system-cni-dir\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.992965 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950914 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-sys\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.950955 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/911301db-2aff-4ec6-ab8a-f613769de8e0-konnectivity-ca\") pod \"konnectivity-agent-b9g5l\" (UID: \"911301db-2aff-4ec6-ab8a-f613769de8e0\") " pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.951000 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35fedfd8-5445-48d3-b505-f22acb89d199-socket-dir\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.951052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-host-run-multus-certs\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.951051 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.951226 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-serviceca\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.951300 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-var-lib-kubelet\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.952648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe2b570d-f7c6-426d-badd-136411182962-os-release\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.957434 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.957444 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.957450 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8b08c62-97d7-4fea-a7ba-54773582ceaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.957439 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a892d681-79c6-4216-b544-5f5db8314362-tmp\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.951306 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a892d681-79c6-4216-b544-5f5db8314362-etc-modprobe-d\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.957556 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a892d681-79c6-4216-b544-5f5db8314362-etc-tuned\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.958496 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-ovn-node-metrics-cert\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:40.993469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.960516 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/911301db-2aff-4ec6-ab8a-f613769de8e0-agent-certs\") pod \"konnectivity-agent-b9g5l\" (UID: \"911301db-2aff-4ec6-ab8a-f613769de8e0\") " pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.965941 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjg6\" (UniqueName: \"kubernetes.io/projected/fe2b570d-f7c6-426d-badd-136411182962-kube-api-access-nvjg6\") pod \"multus-7qrm8\" (UID: \"fe2b570d-f7c6-426d-badd-136411182962\") " pod="openshift-multus/multus-7qrm8" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.966450 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mbj\" (UniqueName: \"kubernetes.io/projected/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-kube-api-access-f6mbj\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.966665 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x624s\" (UniqueName: \"kubernetes.io/projected/35fedfd8-5445-48d3-b505-f22acb89d199-kube-api-access-x624s\") pod \"aws-ebs-csi-driver-node-778gs\" (UID: \"35fedfd8-5445-48d3-b505-f22acb89d199\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.966986 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7h8t\" (UniqueName: \"kubernetes.io/projected/a892d681-79c6-4216-b544-5f5db8314362-kube-api-access-p7h8t\") pod \"tuned-c9bb6\" (UID: \"a892d681-79c6-4216-b544-5f5db8314362\") " pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.967237 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fhv\" (UniqueName: \"kubernetes.io/projected/d8b08c62-97d7-4fea-a7ba-54773582ceaf-kube-api-access-m5fhv\") pod \"multus-additional-cni-plugins-hjpww\" (UID: \"d8b08c62-97d7-4fea-a7ba-54773582ceaf\") " pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.968241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whx9r\" (UniqueName: \"kubernetes.io/projected/e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe-kube-api-access-whx9r\") pod \"ovnkube-node-hhjnx\" (UID: \"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.969171 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.969189 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.969199 2556 projected.go:196] Error preparing data for projected volume kube-api-access-gvcsk for pod openshift-network-diagnostics/network-check-target-b5ssg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:40.969247 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk podName:5545addf-2be8-42e2-a1f6-603d81bed70f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:41.469233669 +0000 UTC m=+2.073748361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gvcsk" (UniqueName: "kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk") pod "network-check-target-b5ssg" (UID: "5545addf-2be8-42e2-a1f6-603d81bed70f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.971016 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfl4\" (UniqueName: \"kubernetes.io/projected/6a19f14e-5a01-44e1-9933-dca850de7aea-kube-api-access-jbfl4\") pod \"node-resolver-ch6ct\" (UID: \"6a19f14e-5a01-44e1-9933-dca850de7aea\") " pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.971737 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85fhs\" (UniqueName: \"kubernetes.io/projected/71faa3e7-1beb-4c83-bce1-a3b45a1e8155-kube-api-access-85fhs\") pod \"node-ca-cj6rf\" (UID: \"71faa3e7-1beb-4c83-bce1-a3b45a1e8155\") " pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:40.994105 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.006959 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.025048 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ch6ct" Jul 09 14:14:41.107998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.050812 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1314eb6f-01a6-472d-8a2b-85128dbe3545-host-slash\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:41.109142 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.050853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtr5q\" (UniqueName: \"kubernetes.io/projected/1314eb6f-01a6-472d-8a2b-85128dbe3545-kube-api-access-wtr5q\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:41.109142 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.050901 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1314eb6f-01a6-472d-8a2b-85128dbe3545-host-slash\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:41.109142 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.050936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1314eb6f-01a6-472d-8a2b-85128dbe3545-iptables-alerter-script\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:41.109142 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.051480 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1314eb6f-01a6-472d-8a2b-85128dbe3545-iptables-alerter-script\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:41.109142 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.074114 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtr5q\" (UniqueName: \"kubernetes.io/projected/1314eb6f-01a6-472d-8a2b-85128dbe3545-kube-api-access-wtr5q\") pod \"iptables-alerter-hjh24\" (UID: \"1314eb6f-01a6-472d-8a2b-85128dbe3545\") " pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:41.205278 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.159285 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" Jul 09 14:14:41.205278 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.177878 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:14:41.205278 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.186753 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cj6rf" Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.209839 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qrm8" Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.228910 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hjpww" Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.258488 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b08c62_97d7_4fea_a7ba_54773582ceaf.slice/crio-94865f067441f56264bc1f1721162feaddfbc09a08923c2a804d93d116d5fb47 WatchSource:0}: Error finding container 94865f067441f56264bc1f1721162feaddfbc09a08923c2a804d93d116d5fb47: Status 404 returned error can't find the container with id 94865f067441f56264bc1f1721162feaddfbc09a08923c2a804d93d116d5fb47 Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.259109 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4097f5baac0499a3289c3b380cb8c1.slice/crio-e57edc397e97d2b08bb4482a1db3241f0107c6e23a7c934fc679b7a028da962e WatchSource:0}: Error finding container e57edc397e97d2b08bb4482a1db3241f0107c6e23a7c934fc679b7a028da962e: Status 404 returned error can't find the container with id e57edc397e97d2b08bb4482a1db3241f0107c6e23a7c934fc679b7a028da962e Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.263572 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.266347 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.290113 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ccd446_9cc6_4a7f_b9ae_4fba80f562fe.slice/crio-ee05956340154b58be2c6960dd6b044ef241da05e56694fd963387f6f1f85ba8 WatchSource:0}: Error finding container ee05956340154b58be2c6960dd6b044ef241da05e56694fd963387f6f1f85ba8: Status 404 returned error can't find the container with id ee05956340154b58be2c6960dd6b044ef241da05e56694fd963387f6f1f85ba8 Jul 09 14:14:41.300925 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.290770 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb67f50f5c5e7503dadf6a1637939a.slice/crio-6c6087c567701d51b7cfcf4c892ad92c9db053c10464e40d9fb4bae602fca5f3 WatchSource:0}: Error finding container 6c6087c567701d51b7cfcf4c892ad92c9db053c10464e40d9fb4bae602fca5f3: Status 404 returned error can't find the container with id 6c6087c567701d51b7cfcf4c892ad92c9db053c10464e40d9fb4bae602fca5f3 Jul 09 14:14:41.427008 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.332958 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hjh24" Jul 09 14:14:41.427008 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.341123 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1314eb6f_01a6_472d_8a2b_85128dbe3545.slice/crio-0f9bf04d83a1e8173f6df9d82f4b1a16dcb867afdbd6f6e73fe56eaebb35b8f3 WatchSource:0}: Error finding container 0f9bf04d83a1e8173f6df9d82f4b1a16dcb867afdbd6f6e73fe56eaebb35b8f3: Status 404 returned error can't find the container with id 0f9bf04d83a1e8173f6df9d82f4b1a16dcb867afdbd6f6e73fe56eaebb35b8f3 Jul 09 14:14:41.454625 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.454598 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:41.454625 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.454628 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:41.454794 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.454742 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:41.454794 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.454785 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:41.454857 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.454796 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret podName:e988dcf9-c3bd-4bc1-95d0-d5217096ec97 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:42.454781317 +0000 UTC m=+3.059295999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret") pod "global-pull-secret-syncer-tmpjn" (UID: "e988dcf9-c3bd-4bc1-95d0-d5217096ec97") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:41.454857 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.454823 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs podName:03bbda19-f3b2-4d1d-8aaa-1c1647b55efa nodeName:}" failed. No retries permitted until 2026-07-09 14:14:42.45481256 +0000 UTC m=+3.059327238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs") pod "network-metrics-daemon-s7f2d" (UID: "03bbda19-f3b2-4d1d-8aaa-1c1647b55efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:41.459795 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.459774 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911301db_2aff_4ec6_ab8a_f613769de8e0.slice/crio-d942787c5389540eb2d6ef1f7a13a3ebc35dd6cab7e1ee94b9833d1be4cef804 WatchSource:0}: Error finding container d942787c5389540eb2d6ef1f7a13a3ebc35dd6cab7e1ee94b9833d1be4cef804: Status 404 returned error can't find the container with id d942787c5389540eb2d6ef1f7a13a3ebc35dd6cab7e1ee94b9833d1be4cef804 Jul 09 14:14:41.494995 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.494973 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda892d681_79c6_4216_b544_5f5db8314362.slice/crio-9f578078204880c020ff7f412c012ba3b666381a3b94480156e296c7647324ce WatchSource:0}: Error finding container 9f578078204880c020ff7f412c012ba3b666381a3b94480156e296c7647324ce: Status 404 returned error can't find the container with id 9f578078204880c020ff7f412c012ba3b666381a3b94480156e296c7647324ce Jul 09 14:14:41.555571 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.555541 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:41.555720 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.555698 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:41.555720 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.555717 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:41.555889 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.555729 2556 projected.go:196] Error preparing data for projected volume kube-api-access-gvcsk for pod openshift-network-diagnostics/network-check-target-b5ssg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:41.555889 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:41.555805 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk podName:5545addf-2be8-42e2-a1f6-603d81bed70f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:42.55576984 +0000 UTC m=+3.160284522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvcsk" (UniqueName: "kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk") pod "network-check-target-b5ssg" (UID: "5545addf-2be8-42e2-a1f6-603d81bed70f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:41.712122 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.712096 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a19f14e_5a01_44e1_9933_dca850de7aea.slice/crio-89eb88f04d80d18ec0004ca0946fcc26b5b3085f11a11e08825392d1fefafeea WatchSource:0}: Error finding container 89eb88f04d80d18ec0004ca0946fcc26b5b3085f11a11e08825392d1fefafeea: Status 404 returned error can't find the container with id 89eb88f04d80d18ec0004ca0946fcc26b5b3085f11a11e08825392d1fefafeea Jul 09 14:14:41.908672 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:14:41.908635 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2b570d_f7c6_426d_badd_136411182962.slice/crio-2c25c3d2dd30e76b1be8b1329341a0082a870b6fcb2354b5cbd900781b708d31 WatchSource:0}: Error finding container 2c25c3d2dd30e76b1be8b1329341a0082a870b6fcb2354b5cbd900781b708d31: Status 404 returned error can't find the container with id 2c25c3d2dd30e76b1be8b1329341a0082a870b6fcb2354b5cbd900781b708d31 Jul 09 14:14:41.919087 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.919020 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:09:40 +0000 UTC" deadline="2028-03-25 23:12:51.56586697 +0000 UTC" Jul 09 14:14:41.919199 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:41.919088 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15008h58m9.646783658s" Jul 09 14:14:42.065757 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.065682 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:14:42.072075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.072050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:42.072221 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.072183 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:42.078467 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.078443 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:14:42.078953 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.078907 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qrm8" event={"ID":"fe2b570d-f7c6-426d-badd-136411182962","Type":"ContainerStarted","Data":"2c25c3d2dd30e76b1be8b1329341a0082a870b6fcb2354b5cbd900781b708d31"} Jul 09 14:14:42.081194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.081119 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ch6ct" event={"ID":"6a19f14e-5a01-44e1-9933-dca850de7aea","Type":"ContainerStarted","Data":"89eb88f04d80d18ec0004ca0946fcc26b5b3085f11a11e08825392d1fefafeea"} Jul 09 14:14:42.083101 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.083038 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hjh24" event={"ID":"1314eb6f-01a6-472d-8a2b-85128dbe3545","Type":"ContainerStarted","Data":"0f9bf04d83a1e8173f6df9d82f4b1a16dcb867afdbd6f6e73fe56eaebb35b8f3"} Jul 09 14:14:42.084469 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.084350 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"ee05956340154b58be2c6960dd6b044ef241da05e56694fd963387f6f1f85ba8"} Jul 09 14:14:42.086910 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.086845 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerStarted","Data":"94865f067441f56264bc1f1721162feaddfbc09a08923c2a804d93d116d5fb47"} Jul 09 14:14:42.088852 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.088784 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" event={"ID":"a892d681-79c6-4216-b544-5f5db8314362","Type":"ContainerStarted","Data":"9f578078204880c020ff7f412c012ba3b666381a3b94480156e296c7647324ce"} Jul 09 14:14:42.090339 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.090274 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b9g5l" event={"ID":"911301db-2aff-4ec6-ab8a-f613769de8e0","Type":"ContainerStarted","Data":"d942787c5389540eb2d6ef1f7a13a3ebc35dd6cab7e1ee94b9833d1be4cef804"} Jul 09 14:14:42.099700 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.099674 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" event={"ID":"23cb67f50f5c5e7503dadf6a1637939a","Type":"ContainerStarted","Data":"6c6087c567701d51b7cfcf4c892ad92c9db053c10464e40d9fb4bae602fca5f3"} Jul 09 14:14:42.102038 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.102015 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" event={"ID":"1b4097f5baac0499a3289c3b380cb8c1","Type":"ContainerStarted","Data":"e57edc397e97d2b08bb4482a1db3241f0107c6e23a7c934fc679b7a028da962e"} Jul 09 14:14:42.461742 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.461706 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:42.461919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.461759 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:42.461978 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.461923 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:42.462030 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.461984 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs podName:03bbda19-f3b2-4d1d-8aaa-1c1647b55efa nodeName:}" failed. No retries permitted until 2026-07-09 14:14:44.461966022 +0000 UTC m=+5.066480702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs") pod "network-metrics-daemon-s7f2d" (UID: "03bbda19-f3b2-4d1d-8aaa-1c1647b55efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:42.462452 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.462415 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:42.462546 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.462492 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret podName:e988dcf9-c3bd-4bc1-95d0-d5217096ec97 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:44.462475839 +0000 UTC m=+5.066990531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret") pod "global-pull-secret-syncer-tmpjn" (UID: "e988dcf9-c3bd-4bc1-95d0-d5217096ec97") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:42.562366 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.562329 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:42.562559 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.562501 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:42.562559 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.562523 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:42.562559 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.562537 2556 projected.go:196] Error preparing data for projected volume kube-api-access-gvcsk for pod openshift-network-diagnostics/network-check-target-b5ssg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:42.562721 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:42.562598 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk podName:5545addf-2be8-42e2-a1f6-603d81bed70f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:44.562579808 +0000 UTC m=+5.167094489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvcsk" (UniqueName: "kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk") pod "network-check-target-b5ssg" (UID: "5545addf-2be8-42e2-a1f6-603d81bed70f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:42.920323 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.920060 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:09:40 +0000 UTC" deadline="2027-12-30 23:48:23.036145925 +0000 UTC" Jul 09 14:14:42.920323 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:42.920293 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12945h33m40.115859419s" Jul 09 14:14:43.070786 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:43.070283 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:43.070786 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:43.070433 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:43.070786 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:43.070449 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:43.070786 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:43.070567 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:43.109168 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:43.109071 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" event={"ID":"35fedfd8-5445-48d3-b505-f22acb89d199","Type":"ContainerStarted","Data":"0905f7b74c1bb7b0625a96e550faa83c55f45e557726d53d722f16e11655e560"} Jul 09 14:14:44.072925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:44.072384 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:44.072925 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.072541 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:44.481336 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:44.481249 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:44.481336 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:44.481300 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:44.481575 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.481414 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:44.481575 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.481415 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:44.481575 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.481493 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs podName:03bbda19-f3b2-4d1d-8aaa-1c1647b55efa nodeName:}" failed. No retries permitted until 2026-07-09 14:14:48.481473912 +0000 UTC m=+9.085988593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs") pod "network-metrics-daemon-s7f2d" (UID: "03bbda19-f3b2-4d1d-8aaa-1c1647b55efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:44.481575 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.481534 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret podName:e988dcf9-c3bd-4bc1-95d0-d5217096ec97 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:48.481516071 +0000 UTC m=+9.086030763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret") pod "global-pull-secret-syncer-tmpjn" (UID: "e988dcf9-c3bd-4bc1-95d0-d5217096ec97") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:44.582616 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:44.582580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:44.582814 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.582770 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:44.582814 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.582795 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:44.582814 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.582809 2556 projected.go:196] Error preparing data for projected volume kube-api-access-gvcsk for pod openshift-network-diagnostics/network-check-target-b5ssg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:44.582985 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:44.582874 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk podName:5545addf-2be8-42e2-a1f6-603d81bed70f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:48.582852979 +0000 UTC m=+9.187367672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvcsk" (UniqueName: "kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk") pod "network-check-target-b5ssg" (UID: "5545addf-2be8-42e2-a1f6-603d81bed70f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:45.070760 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:45.069990 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:45.070760 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:45.070124 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:45.070760 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:45.070240 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:45.070760 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:45.070330 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:46.072230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:46.072199 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:46.072726 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:46.072338 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:47.069737 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:47.069704 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:47.069915 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:47.069840 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:47.069915 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:47.069888 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:47.070038 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:47.070013 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:48.072737 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:48.072706 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:48.073101 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.072860 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:48.512144 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:48.511381 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:48.512144 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:48.511451 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:48.512144 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.511613 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:48.512144 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.511683 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs podName:03bbda19-f3b2-4d1d-8aaa-1c1647b55efa nodeName:}" failed. No retries permitted until 2026-07-09 14:14:56.511662765 +0000 UTC m=+17.116177447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs") pod "network-metrics-daemon-s7f2d" (UID: "03bbda19-f3b2-4d1d-8aaa-1c1647b55efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:48.512144 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.511752 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:48.512144 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.511791 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret podName:e988dcf9-c3bd-4bc1-95d0-d5217096ec97 nodeName:}" failed. No retries permitted until 2026-07-09 14:14:56.511779573 +0000 UTC m=+17.116294258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret") pod "global-pull-secret-syncer-tmpjn" (UID: "e988dcf9-c3bd-4bc1-95d0-d5217096ec97") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:48.612695 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:48.612658 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:48.612876 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.612823 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:48.612876 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.612842 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:48.612876 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.612854 2556 projected.go:196] Error preparing data for projected volume kube-api-access-gvcsk for pod openshift-network-diagnostics/network-check-target-b5ssg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:48.613021 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:48.612900 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk podName:5545addf-2be8-42e2-a1f6-603d81bed70f nodeName:}" failed. No retries permitted until 2026-07-09 14:14:56.612886247 +0000 UTC m=+17.217400925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvcsk" (UniqueName: "kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk") pod "network-check-target-b5ssg" (UID: "5545addf-2be8-42e2-a1f6-603d81bed70f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:49.070273 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:49.070080 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:49.070273 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:49.070115 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:49.070273 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:49.070204 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:49.070628 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:49.070305 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:50.071191 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:50.070705 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:50.071191 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:50.070844 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:51.069436 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:51.069392 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:51.069597 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:51.069539 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:51.069657 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:51.069594 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:51.069746 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:51.069717 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:52.070438 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:52.070041 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:52.070438 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:52.070184 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:52.138644 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:52.137836 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" event={"ID":"a892d681-79c6-4216-b544-5f5db8314362","Type":"ContainerStarted","Data":"5507052bd76232a2d065f379b33f9ffcb6cf58390bc17e66f5dc7fd162a22c46"} Jul 09 14:14:52.143240 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:52.141782 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" event={"ID":"23cb67f50f5c5e7503dadf6a1637939a","Type":"ContainerStarted","Data":"d68fb007a5a194ab2069916fefc79c3553409cad05ed9f98302987aa424d1553"} Jul 09 14:14:52.144002 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:52.143950 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cj6rf" event={"ID":"71faa3e7-1beb-4c83-bce1-a3b45a1e8155","Type":"ContainerStarted","Data":"ebdb552b2cc17172062c754bfc48ff38e2d35691b0625a0dd10be6459a55c2fa"} Jul 09 14:14:52.156357 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:52.155997 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c9bb6" podStartSLOduration=1.756702551 podStartE2EDuration="12.155980109s" totalImagesPullingTime="10.399277558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:52.155701714 +0000 UTC m=+12.760216414" watchObservedRunningTime="2026-07-09 14:14:52.155980109 +0000 UTC m=+12.760494811" Jul 09 14:14:53.069834 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.069600 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:53.070023 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.069600 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:53.070023 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:53.069968 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:53.070160 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:53.070123 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:53.147959 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.147878 2556 generic.go:356] "Generic (PLEG): container finished" podID="1b4097f5baac0499a3289c3b380cb8c1" containerID="d320cc0800e3b0da342646dc62a0d8fb4bf33671c3e02bf1360e570421b7a487" exitCode=0 Jul 09 14:14:53.148515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.147977 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" event={"ID":"1b4097f5baac0499a3289c3b380cb8c1","Type":"ContainerDied","Data":"d320cc0800e3b0da342646dc62a0d8fb4bf33671c3e02bf1360e570421b7a487"} Jul 09 14:14:53.149829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.149801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" event={"ID":"35fedfd8-5445-48d3-b505-f22acb89d199","Type":"ContainerStarted","Data":"e2b58e1af8877514f252c8ca302eea58e25766376fba36b65c066b1266ae7af3"} Jul 09 14:14:53.151442 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.151401 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ch6ct" event={"ID":"6a19f14e-5a01-44e1-9933-dca850de7aea","Type":"ContainerStarted","Data":"8868c8aba718894c7a8fe1ddbd879041c3e5694838cab7699a6b596b192eb65b"} Jul 09 14:14:53.153016 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.152993 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8b08c62-97d7-4fea-a7ba-54773582ceaf" containerID="887f90230932d646100e11ddd2c95d45e09e00eba0eda1b2139a614a43df8984" exitCode=0 Jul 09 14:14:53.153121 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.153074 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerDied","Data":"887f90230932d646100e11ddd2c95d45e09e00eba0eda1b2139a614a43df8984"} Jul 09 14:14:53.154658 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.154631 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b9g5l" event={"ID":"911301db-2aff-4ec6-ab8a-f613769de8e0","Type":"ContainerStarted","Data":"e56150227a7c17641ab1d7f240a4fd5f9c88da8c7fbc8f2f77016462f7a1c1bc"} Jul 09 14:14:53.163044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.162998 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-0.ec2.internal" podStartSLOduration=13.162984688 podStartE2EDuration="13.162984688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:52.169899152 +0000 UTC m=+12.774413852" watchObservedRunningTime="2026-07-09 14:14:53.162984688 +0000 UTC m=+13.767499387" Jul 09 14:14:53.177371 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.177331 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b9g5l" podStartSLOduration=2.76620767 podStartE2EDuration="13.177317412s" totalImagesPullingTime="10.411109742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:53.176781847 +0000 UTC m=+13.781296550" watchObservedRunningTime="2026-07-09 14:14:53.177317412 +0000 UTC m=+13.781832111" Jul 09 14:14:53.196126 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.196083 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ch6ct" podStartSLOduration=3.040227035 podStartE2EDuration="13.196068733s" totalImagesPullingTime="10.155841698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:53.195698289 +0000 UTC m=+13.800212984" watchObservedRunningTime="2026-07-09 14:14:53.196068733 +0000 UTC m=+13.800583435" Jul 09 14:14:53.474363 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.474212 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:53.794203 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.794123 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:53.894211 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:53.894181 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:54.070076 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:54.069991 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:54.070232 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:54.070115 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:54.078708 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:54.078679 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:54.158487 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:54.158391 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hjh24" event={"ID":"1314eb6f-01a6-472d-8a2b-85128dbe3545","Type":"ContainerStarted","Data":"a78c85a6ba707280982db69359b60c38e7dfb1cbbf4e54449ac44f69737be9a7"} Jul 09 14:14:54.173207 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:54.173152 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hjh24" podStartSLOduration=3.64240256 podStartE2EDuration="14.173134436s" totalImagesPullingTime="10.530731876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:14:54.172738477 +0000 UTC m=+14.777253176" watchObservedRunningTime="2026-07-09 14:14:54.173134436 +0000 UTC m=+14.777649136" Jul 09 14:14:54.443103 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:54.443023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:55.069901 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.069872 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:55.070092 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.069877 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:55.070092 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:55.070006 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:55.070092 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:55.070051 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:55.279147 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.279113 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:55.348452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.348359 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:55.348922 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.348896 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:14:55.479396 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.479365 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:55.682920 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.682835 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:55.883937 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:55.883897 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:56.069679 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:56.069599 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:56.069830 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.069746 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:56.094061 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:56.094030 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:56.547065 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:56.546979 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:56.581050 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:56.581000 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:56.581219 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:56.581070 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:56.581219 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.581153 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:56.581219 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.581209 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret podName:e988dcf9-c3bd-4bc1-95d0-d5217096ec97 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:12.581192835 +0000 UTC m=+33.185707512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret") pod "global-pull-secret-syncer-tmpjn" (UID: "e988dcf9-c3bd-4bc1-95d0-d5217096ec97") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:14:56.581363 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.581221 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:56.581363 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.581278 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs podName:03bbda19-f3b2-4d1d-8aaa-1c1647b55efa nodeName:}" failed. No retries permitted until 2026-07-09 14:15:12.581254345 +0000 UTC m=+33.185769037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs") pod "network-metrics-daemon-s7f2d" (UID: "03bbda19-f3b2-4d1d-8aaa-1c1647b55efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:14:56.682030 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:56.682004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:56.682177 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.682140 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:14:56.682177 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.682157 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:14:56.682177 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.682175 2556 projected.go:196] Error preparing data for projected volume kube-api-access-gvcsk for pod openshift-network-diagnostics/network-check-target-b5ssg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:56.682324 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:56.682220 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk podName:5545addf-2be8-42e2-a1f6-603d81bed70f nodeName:}" failed. No retries permitted until 2026-07-09 14:15:12.6822076 +0000 UTC m=+33.286722277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvcsk" (UniqueName: "kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk") pod "network-check-target-b5ssg" (UID: "5545addf-2be8-42e2-a1f6-603d81bed70f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:14:56.905814 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:56.905711 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:14:57.069858 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:57.069772 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:57.069858 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:57.069820 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:57.070080 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:57.069913 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:57.070080 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:57.070034 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:14:58.070066 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:58.070033 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:14:58.070558 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:58.070187 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:14:59.070270 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:59.070236 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:14:59.070768 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:14:59.070236 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:14:59.070768 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:59.070367 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:14:59.070768 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:14:59.070410 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:00.071150 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:00.070957 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:00.071719 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:00.071250 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:00.233389 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:00.233346 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:00.292505 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:00.292466 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:01.069788 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:01.069752 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:01.069973 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:01.069797 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:01.069973 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:01.069862 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:01.070094 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:01.070066 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:02.070143 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:02.069926 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:02.070143 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:02.070066 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:03.070125 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:03.070090 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:03.070454 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:03.070206 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:03.070454 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:03.070265 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:03.070454 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:03.070381 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:04.069661 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:04.069626 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:04.069814 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:04.069745 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:04.281652 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:04.281626 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 14:15:04.922521 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:04.922254 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T14:15:04.281648803Z","UUID":"af730ca1-d5a6-44d6-b8c1-50f078850e3b","Handler":null,"Name":"","Endpoint":""} Jul 09 14:15:04.923600 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:04.923579 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 14:15:04.923717 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:04.923612 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 14:15:05.070352 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.070313 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:05.070535 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:05.070461 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:05.070595 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.070524 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:05.070671 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:05.070639 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:05.181639 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.181574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" event={"ID":"1b4097f5baac0499a3289c3b380cb8c1","Type":"ContainerStarted","Data":"d1056c2c476cfe71581f586c6a6d44593c21a13b24744cdfdb3d670d88e6f932"} Jul 09 14:15:05.183332 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.183308 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" event={"ID":"35fedfd8-5445-48d3-b505-f22acb89d199","Type":"ContainerStarted","Data":"23578c74aada0f2ac350e87db249e27bd5aa3b720ead1a084f6d6eb29ba0e60f"} Jul 09 14:15:05.184557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.184537 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cj6rf" event={"ID":"71faa3e7-1beb-4c83-bce1-a3b45a1e8155","Type":"ContainerStarted","Data":"6d42afdce1731d9027704715751aeb7b19a4dd2a65e15b7d3f88a2c2a3af7abc"} Jul 09 14:15:05.185919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.185900 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qrm8" event={"ID":"fe2b570d-f7c6-426d-badd-136411182962","Type":"ContainerStarted","Data":"d456c09d605719deb6983de5108287847851177e3520904cec23bdddd74fa7c6"} Jul 09 14:15:05.188385 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.188369 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:15:05.188701 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.188676 2556 generic.go:356] "Generic (PLEG): container finished" podID="e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe" containerID="4468f0397b423d49ae83c6d910723853db25ad9d63bcc52808dcc5cd9e336b32" exitCode=1 Jul 09 14:15:05.188755 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.188742 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"4e8051c7c31742f945d232df4b25a67743c5658f0c7fbaa60d2c430e19de861e"} Jul 09 14:15:05.188877 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.188858 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"9d038688c9ffc779da2be871877dfac77b3100ab7e566c05f2154783b8801f69"} Jul 09 14:15:05.188929 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.188916 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"a338715dead63eda3626fe9cd50af641f94670892bbb79f0b2d3e9d1ff94ae77"} Jul 09 14:15:05.188974 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.188952 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"2a12bb0aa360053f68160351954311b516b3237390cda8db7532941df8006a3a"} Jul 09 14:15:05.189017 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.188991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerDied","Data":"4468f0397b423d49ae83c6d910723853db25ad9d63bcc52808dcc5cd9e336b32"} Jul 09 14:15:05.189053 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.189028 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"24cc75a317329d46fb9f29dd649ec17dea26b85bd62e2a02ad4d3adb8fd1ac21"} Jul 09 14:15:05.190379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.190360 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8b08c62-97d7-4fea-a7ba-54773582ceaf" containerID="6a03b8864dd86615fe5bd351b71a970c38afe57b28b424185fddef1480515fa8" exitCode=0 Jul 09 14:15:05.190468 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.190385 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerDied","Data":"6a03b8864dd86615fe5bd351b71a970c38afe57b28b424185fddef1480515fa8"} Jul 09 14:15:05.197175 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.197140 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-0.ec2.internal" podStartSLOduration=25.197127602 podStartE2EDuration="25.197127602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:05.196541233 +0000 UTC m=+25.801055954" watchObservedRunningTime="2026-07-09 14:15:05.197127602 +0000 UTC m=+25.801642298" Jul 09 14:15:05.232871 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:05.232836 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7qrm8" podStartSLOduration=4.981709115 podStartE2EDuration="25.232824776s" totalImagesPullingTime="20.251115661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:05.232649476 +0000 UTC m=+25.837164177" watchObservedRunningTime="2026-07-09 14:15:05.232824776 +0000 UTC m=+25.837339475" Jul 09 14:15:06.070473 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.070445 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:06.071049 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:06.070587 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:06.119447 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.119406 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:06.194928 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.194889 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" event={"ID":"35fedfd8-5445-48d3-b505-f22acb89d199","Type":"ContainerStarted","Data":"64fb62c34a370f4b1f5edbda92b5a2116c34ca475772ef58899f66d5f98ee1a2"} Jul 09 14:15:06.231231 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.231189 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:06.232779 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.232745 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cj6rf" podStartSLOduration=15.954769818 podStartE2EDuration="26.232732039s" totalImagesPullingTime="10.277962221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:05.245563659 +0000 UTC m=+25.850078360" watchObservedRunningTime="2026-07-09 14:15:06.232732039 +0000 UTC m=+26.837246757" Jul 09 14:15:06.233092 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.233073 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-778gs" podStartSLOduration=3.416706025 podStartE2EDuration="26.233068306s" totalImagesPullingTime="22.816362281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:06.232211871 +0000 UTC m=+26.836726571" watchObservedRunningTime="2026-07-09 14:15:06.233068306 +0000 UTC m=+26.837583006" Jul 09 14:15:06.373309 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.373236 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:06.462756 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.462726 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:06.556233 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.556202 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:06.690313 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.690284 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:06.794019 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:06.793985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:07.069485 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:07.069452 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:07.069700 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:07.069452 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:07.069700 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:07.069580 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:07.069700 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:07.069649 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:07.108759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:07.108729 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:07.199381 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:07.199324 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:15:07.199685 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:07.199663 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"74a99c2ec86100879d95398c657613aa9217cb9c78f7ceab5aab9a33f73c34f1"} Jul 09 14:15:07.201711 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:07.201681 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8b08c62-97d7-4fea-a7ba-54773582ceaf" containerID="348f9eb814c441958284d5f4512fe6637789d3abe0fd1ccdeffa8f7d6d45eb02" exitCode=0 Jul 09 14:15:07.201793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:07.201762 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerDied","Data":"348f9eb814c441958284d5f4512fe6637789d3abe0fd1ccdeffa8f7d6d45eb02"} Jul 09 14:15:08.069617 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:08.069591 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:08.069761 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:08.069696 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:09.070340 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.070213 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:09.070641 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.070213 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:09.070641 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:09.070405 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:09.070641 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:09.070496 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:09.209052 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.209029 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:15:09.209416 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.209388 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"80880caa94282e0f0ee7ae5ad3e8521b83923b0d064f919553284f55f22e40ae"} Jul 09 14:15:09.209683 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.209665 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:15:09.209879 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.209861 2556 scope.go:117] "RemoveContainer" containerID="4468f0397b423d49ae83c6d910723853db25ad9d63bcc52808dcc5cd9e336b32" Jul 09 14:15:09.211990 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.211962 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8b08c62-97d7-4fea-a7ba-54773582ceaf" containerID="551c71bbe237ae3be06d59fe4d49e566d248ffc878124e2f86263e161b90d4d8" exitCode=0 Jul 09 14:15:09.212083 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.211996 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerDied","Data":"551c71bbe237ae3be06d59fe4d49e566d248ffc878124e2f86263e161b90d4d8"} Jul 09 14:15:09.226653 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.226625 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:15:09.755673 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.755645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:09.855019 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:09.854978 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:10.070383 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.070354 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:10.070783 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:10.070464 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:10.217228 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.217201 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:15:10.217585 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.217567 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" event={"ID":"e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe","Type":"ContainerStarted","Data":"b6b0d19f237708ddec955469dcbec90ea8afc95bc5d1152b96a7c7d12791f3da"} Jul 09 14:15:10.217752 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.217729 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:15:10.217870 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.217785 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:15:10.233247 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.233225 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:15:10.245160 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.245118 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" podStartSLOduration=7.435080129 podStartE2EDuration="30.245105228s" totalImagesPullingTime="22.810025099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:10.243862337 +0000 UTC m=+30.848377036" watchObservedRunningTime="2026-07-09 14:15:10.245105228 +0000 UTC m=+30.849619928" Jul 09 14:15:10.908767 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.908732 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:15:10.908925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.908909 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 14:15:10.909616 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.909589 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b9g5l" Jul 09 14:15:10.971062 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:10.971031 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:11.070453 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.070411 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:11.070819 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:11.070539 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:11.070819 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.070435 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:11.070819 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:11.070617 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:11.293761 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.293487 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5ssg"] Jul 09 14:15:11.293905 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.293803 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:11.293976 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:11.293908 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:11.297158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.297136 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tmpjn"] Jul 09 14:15:11.297271 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.297213 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:11.297320 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:11.297296 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:11.297881 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.297858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s7f2d"] Jul 09 14:15:11.298084 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:11.297951 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:11.298084 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:11.298050 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:12.601748 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:12.601711 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:12.602185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:12.601810 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:12.602185 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.601858 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:15:12.602185 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.601913 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:15:12.602185 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.601920 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs podName:03bbda19-f3b2-4d1d-8aaa-1c1647b55efa nodeName:}" failed. No retries permitted until 2026-07-09 14:15:44.601906913 +0000 UTC m=+65.206421591 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs") pod "network-metrics-daemon-s7f2d" (UID: "03bbda19-f3b2-4d1d-8aaa-1c1647b55efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:15:12.602185 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.601975 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret podName:e988dcf9-c3bd-4bc1-95d0-d5217096ec97 nodeName:}" failed. No retries permitted until 2026-07-09 14:15:44.6019585 +0000 UTC m=+65.206473179 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret") pod "global-pull-secret-syncer-tmpjn" (UID: "e988dcf9-c3bd-4bc1-95d0-d5217096ec97") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:15:12.702454 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:12.702412 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:12.702598 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.702559 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:15:12.702598 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.702576 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:15:12.702598 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.702585 2556 projected.go:196] Error preparing data for projected volume kube-api-access-gvcsk for pod openshift-network-diagnostics/network-check-target-b5ssg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:15:12.702788 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:12.702635 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk podName:5545addf-2be8-42e2-a1f6-603d81bed70f nodeName:}" failed. No retries permitted until 2026-07-09 14:15:44.702623132 +0000 UTC m=+65.307137815 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvcsk" (UniqueName: "kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk") pod "network-check-target-b5ssg" (UID: "5545addf-2be8-42e2-a1f6-603d81bed70f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:15:13.070125 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:13.070092 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:13.070284 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:13.070105 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:13.070284 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:13.070230 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:13.070284 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:13.070107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:13.070395 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:13.070283 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:13.070395 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:13.070348 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:15.069829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:15.069799 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:15.069829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:15.069818 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:15.070368 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:15.069914 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:15.070368 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:15.070009 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:15.070368 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:15.070035 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:15.070368 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:15.070140 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:15.602741 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:15.602695 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:15.702121 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:15.702086 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:17.070231 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.070194 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:17.070838 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:17.070302 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmpjn" podUID="e988dcf9-c3bd-4bc1-95d0-d5217096ec97" Jul 09 14:15:17.070838 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.070194 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:17.070838 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.070196 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:17.070838 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:17.070392 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s7f2d" podUID="03bbda19-f3b2-4d1d-8aaa-1c1647b55efa" Jul 09 14:15:17.070838 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:17.070455 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5ssg" podUID="5545addf-2be8-42e2-a1f6-603d81bed70f" Jul 09 14:15:17.236528 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.236496 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8b08c62-97d7-4fea-a7ba-54773582ceaf" containerID="7b6a50012a12db9e1d0cbd8c5354fd650fcd5c7de81f043c1e43c23ba6faf188" exitCode=0 Jul 09 14:15:17.236689 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.236547 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerDied","Data":"7b6a50012a12db9e1d0cbd8c5354fd650fcd5c7de81f043c1e43c23ba6faf188"} Jul 09 14:15:17.691016 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.690921 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-0.ec2.internal" event="NodeReady" Jul 09 14:15:17.691122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.691038 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 14:15:17.698640 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.698612 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:17.737546 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.737522 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vcf89"] Jul 09 14:15:17.754828 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.754811 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nd8dl"] Jul 09 14:15:17.754941 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.754925 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:17.758095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.758075 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 14:15:17.758720 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.758698 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 14:15:17.758839 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.758814 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4fkhx\"" Jul 09 14:15:17.758929 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.758908 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 14:15:17.778919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.778898 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vcf89"] Jul 09 14:15:17.779034 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.778930 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nd8dl"] Jul 09 14:15:17.779034 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.778967 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c2lkd"] Jul 09 14:15:17.779133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.779095 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:17.781362 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.781342 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 14:15:17.781514 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.781491 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 14:15:17.781587 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.781512 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 14:15:17.781647 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.781590 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 14:15:17.781710 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.781693 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dj2q5\"" Jul 09 14:15:17.803030 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.803013 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2lkd"] Jul 09 14:15:17.803121 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.803104 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:17.805350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.805326 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 14:15:17.805350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.805335 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:17.805541 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.805527 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 14:15:17.805583 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.805533 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kggvr\"" Jul 09 14:15:17.901251 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.901221 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:17.937368 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fz59\" (UniqueName: \"kubernetes.io/projected/df2b62f9-61e7-44ed-8880-3772033e8e9c-kube-api-access-9fz59\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:17.937483 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937377 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1aae7da-8e11-42fc-b740-ee4e35e26793-crio-socket\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:17.937483 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937405 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1aae7da-8e11-42fc-b740-ee4e35e26793-data-volume\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:17.937587 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937488 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b62f9-61e7-44ed-8880-3772033e8e9c-config-volume\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:17.937587 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937509 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2b62f9-61e7-44ed-8880-3772033e8e9c-metrics-tls\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:17.937587 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3ea1425-a7d9-4581-b20e-44e0560c8cae-cert\") pod \"ingress-canary-vcf89\" (UID: \"c3ea1425-a7d9-4581-b20e-44e0560c8cae\") " pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:17.937730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937608 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwszw\" (UniqueName: \"kubernetes.io/projected/c3ea1425-a7d9-4581-b20e-44e0560c8cae-kube-api-access-mwszw\") pod \"ingress-canary-vcf89\" (UID: \"c3ea1425-a7d9-4581-b20e-44e0560c8cae\") " pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:17.937730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937636 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1aae7da-8e11-42fc-b740-ee4e35e26793-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:17.937730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khw5q\" (UniqueName: \"kubernetes.io/projected/e1aae7da-8e11-42fc-b740-ee4e35e26793-kube-api-access-khw5q\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:17.937730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937690 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b62f9-61e7-44ed-8880-3772033e8e9c-tmp-dir\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:17.937730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.937719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1aae7da-8e11-42fc-b740-ee4e35e26793-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:17.958441 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:17.958374 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:18.037983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.037958 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fz59\" (UniqueName: \"kubernetes.io/projected/df2b62f9-61e7-44ed-8880-3772033e8e9c-kube-api-access-9fz59\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.038103 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.037994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1aae7da-8e11-42fc-b740-ee4e35e26793-crio-socket\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.038103 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1aae7da-8e11-42fc-b740-ee4e35e26793-data-volume\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.038103 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b62f9-61e7-44ed-8880-3772033e8e9c-config-volume\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.038103 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038078 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2b62f9-61e7-44ed-8880-3772033e8e9c-metrics-tls\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.038292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038102 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3ea1425-a7d9-4581-b20e-44e0560c8cae-cert\") pod \"ingress-canary-vcf89\" (UID: \"c3ea1425-a7d9-4581-b20e-44e0560c8cae\") " pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:18.038292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038178 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1aae7da-8e11-42fc-b740-ee4e35e26793-crio-socket\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.038292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwszw\" (UniqueName: \"kubernetes.io/projected/c3ea1425-a7d9-4581-b20e-44e0560c8cae-kube-api-access-mwszw\") pod \"ingress-canary-vcf89\" (UID: \"c3ea1425-a7d9-4581-b20e-44e0560c8cae\") " pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:18.038292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038248 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1aae7da-8e11-42fc-b740-ee4e35e26793-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.038292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khw5q\" (UniqueName: \"kubernetes.io/projected/e1aae7da-8e11-42fc-b740-ee4e35e26793-kube-api-access-khw5q\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.038292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038290 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b62f9-61e7-44ed-8880-3772033e8e9c-tmp-dir\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.038681 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038319 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1aae7da-8e11-42fc-b740-ee4e35e26793-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.038730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038688 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df2b62f9-61e7-44ed-8880-3772033e8e9c-config-volume\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.038730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038699 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b62f9-61e7-44ed-8880-3772033e8e9c-tmp-dir\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.038835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038808 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1aae7da-8e11-42fc-b740-ee4e35e26793-data-volume\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.038908 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.038891 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1aae7da-8e11-42fc-b740-ee4e35e26793-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.041715 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.041695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2b62f9-61e7-44ed-8880-3772033e8e9c-metrics-tls\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.041799 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.041725 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1aae7da-8e11-42fc-b740-ee4e35e26793-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.041799 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.041743 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3ea1425-a7d9-4581-b20e-44e0560c8cae-cert\") pod \"ingress-canary-vcf89\" (UID: \"c3ea1425-a7d9-4581-b20e-44e0560c8cae\") " pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:18.045851 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.045830 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khw5q\" (UniqueName: \"kubernetes.io/projected/e1aae7da-8e11-42fc-b740-ee4e35e26793-kube-api-access-khw5q\") pod \"insights-runtime-extractor-nd8dl\" (UID: \"e1aae7da-8e11-42fc-b740-ee4e35e26793\") " pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.046045 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.046026 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwszw\" (UniqueName: \"kubernetes.io/projected/c3ea1425-a7d9-4581-b20e-44e0560c8cae-kube-api-access-mwszw\") pod \"ingress-canary-vcf89\" (UID: \"c3ea1425-a7d9-4581-b20e-44e0560c8cae\") " pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:18.046399 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.046376 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fz59\" (UniqueName: \"kubernetes.io/projected/df2b62f9-61e7-44ed-8880-3772033e8e9c-kube-api-access-9fz59\") pod \"dns-default-c2lkd\" (UID: \"df2b62f9-61e7-44ed-8880-3772033e8e9c\") " pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.064989 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.064970 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vcf89" Jul 09 14:15:18.087831 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.087813 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nd8dl" Jul 09 14:15:18.112009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.111991 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:18.251095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.250543 2556 generic.go:356] "Generic (PLEG): container finished" podID="d8b08c62-97d7-4fea-a7ba-54773582ceaf" containerID="1afdcd95791855a2953dbb77504fe048005000818102562411571b41c24f7f15" exitCode=0 Jul 09 14:15:18.251095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.250805 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerDied","Data":"1afdcd95791855a2953dbb77504fe048005000818102562411571b41c24f7f15"} Jul 09 14:15:18.280957 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.280931 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nd8dl"] Jul 09 14:15:18.285649 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:18.285626 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1aae7da_8e11_42fc_b740_ee4e35e26793.slice/crio-6cbee6f6cc5eb09fe0001ba0f7e3b032bf9a340327660a6b7ee87456ea2b8365 WatchSource:0}: Error finding container 6cbee6f6cc5eb09fe0001ba0f7e3b032bf9a340327660a6b7ee87456ea2b8365: Status 404 returned error can't find the container with id 6cbee6f6cc5eb09fe0001ba0f7e3b032bf9a340327660a6b7ee87456ea2b8365 Jul 09 14:15:18.293215 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.293148 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2lkd"] Jul 09 14:15:18.295944 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:18.295918 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vcf89"] Jul 09 14:15:18.297214 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:18.297187 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2b62f9_61e7_44ed_8880_3772033e8e9c.slice/crio-2da6a100ca5d5f2ba699e2fa21efdeb34cf2b2cd33cb4a7a90de6469de889d4c WatchSource:0}: Error finding container 2da6a100ca5d5f2ba699e2fa21efdeb34cf2b2cd33cb4a7a90de6469de889d4c: Status 404 returned error can't find the container with id 2da6a100ca5d5f2ba699e2fa21efdeb34cf2b2cd33cb4a7a90de6469de889d4c Jul 09 14:15:18.300442 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:18.300358 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ea1425_a7d9_4581_b20e_44e0560c8cae.slice/crio-a8c83dd6db2199127d134d41cba492b40b584cd4b25d61e665c16de38e36133e WatchSource:0}: Error finding container a8c83dd6db2199127d134d41cba492b40b584cd4b25d61e665c16de38e36133e: Status 404 returned error can't find the container with id a8c83dd6db2199127d134d41cba492b40b584cd4b25d61e665c16de38e36133e Jul 09 14:15:19.069622 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.069587 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:19.069622 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.069617 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:19.069860 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.069629 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:19.076050 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.076024 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cs7kw\"" Jul 09 14:15:19.076317 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.076298 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 14:15:19.076554 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.076535 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 14:15:19.079094 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.079073 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 14:15:19.079288 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.079272 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wzd4v\"" Jul 09 14:15:19.079557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.079539 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 14:15:19.261462 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.261408 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hjpww" event={"ID":"d8b08c62-97d7-4fea-a7ba-54773582ceaf","Type":"ContainerStarted","Data":"8085b6cbcb43ff612d1beaadd7c2ee7395a92ea558f4c59cef7926a88192c48f"} Jul 09 14:15:19.264792 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.264761 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2lkd" event={"ID":"df2b62f9-61e7-44ed-8880-3772033e8e9c","Type":"ContainerStarted","Data":"2da6a100ca5d5f2ba699e2fa21efdeb34cf2b2cd33cb4a7a90de6469de889d4c"} Jul 09 14:15:19.267679 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.267651 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd8dl" event={"ID":"e1aae7da-8e11-42fc-b740-ee4e35e26793","Type":"ContainerStarted","Data":"6cbee6f6cc5eb09fe0001ba0f7e3b032bf9a340327660a6b7ee87456ea2b8365"} Jul 09 14:15:19.270305 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.270282 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vcf89" event={"ID":"c3ea1425-a7d9-4581-b20e-44e0560c8cae","Type":"ContainerStarted","Data":"a8c83dd6db2199127d134d41cba492b40b584cd4b25d61e665c16de38e36133e"} Jul 09 14:15:19.427121 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.426794 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:19.497393 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:19.497324 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:20.689853 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:20.689816 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:20.897139 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:20.897047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:20.936880 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:20.936789 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hjpww" podStartSLOduration=9.944364431 podStartE2EDuration="40.936767792s" totalImagesPullingTime="30.992403361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:19.295665253 +0000 UTC m=+39.900179965" watchObservedRunningTime="2026-07-09 14:15:20.936767792 +0000 UTC m=+41.541282492" Jul 09 14:15:20.937465 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:20.937442 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7"] Jul 09 14:15:20.999561 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:20.999527 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:21.021973 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.021942 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7"] Jul 09 14:15:21.022133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.021999 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-f8htr"] Jul 09 14:15:21.022133 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.022051 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.026044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.026025 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 14:15:21.026392 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.026377 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 14:15:21.026626 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.026604 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bc8rf\"" Jul 09 14:15:21.026698 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.026623 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 14:15:21.027037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.027018 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 14:15:21.027594 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.027579 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 14:15:21.056098 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.056066 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-f8htr"] Jul 09 14:15:21.056245 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.056117 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pktfk"] Jul 09 14:15:21.056245 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.056199 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.058808 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.058778 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 14:15:21.058914 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.058789 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 14:15:21.058914 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.058871 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6bp6x\"" Jul 09 14:15:21.059030 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.058961 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 14:15:21.073636 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.073601 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:21.097564 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.097537 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.100051 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.100029 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 14:15:21.100171 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.100080 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fjdmz\"" Jul 09 14:15:21.100171 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.100031 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 14:15:21.100171 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.100146 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 14:15:21.166671 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.166587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5cl\" (UniqueName: \"kubernetes.io/projected/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-api-access-6b5cl\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.166671 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.166648 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.166848 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.166724 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fda6cf0f-675a-48a5-9b8d-b2e390b37031-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.166848 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.166752 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrnf\" (UniqueName: \"kubernetes.io/projected/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-kube-api-access-9xrnf\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.166848 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.166776 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fda6cf0f-675a-48a5-9b8d-b2e390b37031-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.166951 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.166934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.166990 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.166973 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.167043 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.167012 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.167084 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.167047 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.167124 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.167081 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.196663 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.196623 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:21.267996 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.267963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5cl\" (UniqueName: \"kubernetes.io/projected/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-api-access-6b5cl\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.268162 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-tls\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268162 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fda6cf0f-675a-48a5-9b8d-b2e390b37031-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.268162 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-wtmp\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268308 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268225 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.268308 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268268 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-accelerators-collector-config\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268308 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268304 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrnf\" (UniqueName: \"kubernetes.io/projected/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-kube-api-access-9xrnf\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.268463 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268332 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fda6cf0f-675a-48a5-9b8d-b2e390b37031-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.268463 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268364 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b50e9c2-964e-4a44-a882-7476b118808f-metrics-client-ca\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268463 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-textfile\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268467 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.268557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268473 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fda6cf0f-675a-48a5-9b8d-b2e390b37031-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.268557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268494 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.268557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268533 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.268672 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.268672 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268605 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-sys\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268672 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268629 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-root\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268672 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268664 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268814 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268686 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8t6\" (UniqueName: \"kubernetes.io/projected/9b50e9c2-964e-4a44-a882-7476b118808f-kube-api-access-ws8t6\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.268814 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.268738 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.268917 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:21.268826 2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 09 14:15:21.268965 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:15:21.268916 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-tls podName:591f3a7b-19a2-4ad1-ae4b-39b7e852811b nodeName:}" failed. No retries permitted until 2026-07-09 14:15:21.768895962 +0000 UTC m=+42.373410654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-sfmh7" (UID: "591f3a7b-19a2-4ad1-ae4b-39b7e852811b") : secret "openshift-state-metrics-tls" not found Jul 09 14:15:21.269541 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.269471 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fda6cf0f-675a-48a5-9b8d-b2e390b37031-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.269541 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.269494 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.269862 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.269838 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.273181 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.273156 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.273295 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.273230 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.273927 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.273908 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.281393 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.281324 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5cl\" (UniqueName: \"kubernetes.io/projected/fda6cf0f-675a-48a5-9b8d-b2e390b37031-kube-api-access-6b5cl\") pod \"kube-state-metrics-785778ff74-f8htr\" (UID: \"fda6cf0f-675a-48a5-9b8d-b2e390b37031\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.282577 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.282558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrnf\" (UniqueName: \"kubernetes.io/projected/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-kube-api-access-9xrnf\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.298586 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.298334 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:21.366018 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.366001 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" Jul 09 14:15:21.369249 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369226 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369341 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369252 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8t6\" (UniqueName: \"kubernetes.io/projected/9b50e9c2-964e-4a44-a882-7476b118808f-kube-api-access-ws8t6\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369723 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369703 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-tls\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369789 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369753 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-wtmp\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369789 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369783 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-accelerators-collector-config\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369882 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369817 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b50e9c2-964e-4a44-a882-7476b118808f-metrics-client-ca\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369882 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-textfile\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369928 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-sys\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.369983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.369954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-root\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.370378 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.370347 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-root\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.370477 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.370452 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-sys\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.370603 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.370583 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-wtmp\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.371104 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.370832 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-textfile\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.371204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.371183 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b50e9c2-964e-4a44-a882-7476b118808f-metrics-client-ca\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.371446 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.370801 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-accelerators-collector-config\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.374091 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.373592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.374091 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.373709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b50e9c2-964e-4a44-a882-7476b118808f-node-exporter-tls\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.378813 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.378789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8t6\" (UniqueName: \"kubernetes.io/projected/9b50e9c2-964e-4a44-a882-7476b118808f-kube-api-access-ws8t6\") pod \"node-exporter-pktfk\" (UID: \"9b50e9c2-964e-4a44-a882-7476b118808f\") " pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.401461 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.401413 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:21.407685 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.407671 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pktfk" Jul 09 14:15:21.499540 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.499506 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:21.599121 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.599093 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:21.773624 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.773540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.776603 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.776575 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/591f3a7b-19a2-4ad1-ae4b-39b7e852811b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-sfmh7\" (UID: \"591f3a7b-19a2-4ad1-ae4b-39b7e852811b\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.932780 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.932748 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" Jul 09 14:15:21.954702 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:21.954646 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:23.652534 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:23.652486 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b50e9c2_964e_4a44_a882_7476b118808f.slice/crio-9c182fd46a6e9a7db8fe905001aae21fc059da42f75407049ef17445997cefe7 WatchSource:0}: Error finding container 9c182fd46a6e9a7db8fe905001aae21fc059da42f75407049ef17445997cefe7: Status 404 returned error can't find the container with id 9c182fd46a6e9a7db8fe905001aae21fc059da42f75407049ef17445997cefe7 Jul 09 14:15:23.792592 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.792358 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7"] Jul 09 14:15:23.795845 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.795775 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-f8htr"] Jul 09 14:15:23.839550 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.839502 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:23.916575 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.916499 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59478f8d8f-5h4rw"] Jul 09 14:15:23.923731 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.923702 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:23.926115 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.926091 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mb22p\"" Jul 09 14:15:23.926115 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.926107 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 14:15:23.926302 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.926107 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 14:15:23.926302 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.926214 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 14:15:23.926837 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.926815 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 14:15:23.927027 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.927008 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 14:15:23.927991 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.927970 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 14:15:23.928101 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.928008 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 14:15:23.929643 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.929622 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59478f8d8f-5h4rw"] Jul 09 14:15:23.989280 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.989244 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-console-config\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:23.989456 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.989365 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-serving-cert\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:23.989456 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.989392 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-oauth-serving-cert\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:23.989456 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.989416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-oauth-config\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:23.989621 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.989534 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfrx9\" (UniqueName: \"kubernetes.io/projected/6995c392-15cf-4a9b-8837-622f3f360654-kube-api-access-dfrx9\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:23.989621 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:23.989582 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-service-ca\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.090143 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.090111 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfrx9\" (UniqueName: \"kubernetes.io/projected/6995c392-15cf-4a9b-8837-622f3f360654-kube-api-access-dfrx9\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.090320 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.090165 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-service-ca\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.090320 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.090201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-console-config\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.090320 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.090291 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-serving-cert\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.090320 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.090315 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-oauth-serving-cert\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.090553 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.090344 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-oauth-config\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.091078 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.091045 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-console-config\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.091078 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.091053 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-service-ca\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.091438 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.091402 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-oauth-serving-cert\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.093759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.093734 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-serving-cert\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.093878 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.093858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-oauth-config\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.098618 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.098593 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfrx9\" (UniqueName: \"kubernetes.io/projected/6995c392-15cf-4a9b-8837-622f3f360654-kube-api-access-dfrx9\") pod \"console-59478f8d8f-5h4rw\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.235638 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.235571 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:24.239535 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:24.239504 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591f3a7b_19a2_4ad1_ae4b_39b7e852811b.slice/crio-fa0375d9280ac8c4dd37a4265ae743464e4f4e5df11b532de3ae1cf900e6f840 WatchSource:0}: Error finding container fa0375d9280ac8c4dd37a4265ae743464e4f4e5df11b532de3ae1cf900e6f840: Status 404 returned error can't find the container with id fa0375d9280ac8c4dd37a4265ae743464e4f4e5df11b532de3ae1cf900e6f840 Jul 09 14:15:24.240111 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:24.240088 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda6cf0f_675a_48a5_9b8d_b2e390b37031.slice/crio-b8b9263ee4fa7ef7ffcc4b20d566c50c5658ab7c5db5395781bf3db2be024b9f WatchSource:0}: Error finding container b8b9263ee4fa7ef7ffcc4b20d566c50c5658ab7c5db5395781bf3db2be024b9f: Status 404 returned error can't find the container with id b8b9263ee4fa7ef7ffcc4b20d566c50c5658ab7c5db5395781bf3db2be024b9f Jul 09 14:15:24.284957 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.284895 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" event={"ID":"fda6cf0f-675a-48a5-9b8d-b2e390b37031","Type":"ContainerStarted","Data":"b8b9263ee4fa7ef7ffcc4b20d566c50c5658ab7c5db5395781bf3db2be024b9f"} Jul 09 14:15:24.286523 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.286499 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vcf89" event={"ID":"c3ea1425-a7d9-4581-b20e-44e0560c8cae","Type":"ContainerStarted","Data":"2a4a682aa6e0594c2decb5c82928ea313a0ed558cb61cd6a3bcc3c74b2372703"} Jul 09 14:15:24.288049 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.287989 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" event={"ID":"591f3a7b-19a2-4ad1-ae4b-39b7e852811b","Type":"ContainerStarted","Data":"fa0375d9280ac8c4dd37a4265ae743464e4f4e5df11b532de3ae1cf900e6f840"} Jul 09 14:15:24.289447 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.289407 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pktfk" event={"ID":"9b50e9c2-964e-4a44-a882-7476b118808f","Type":"ContainerStarted","Data":"9c182fd46a6e9a7db8fe905001aae21fc059da42f75407049ef17445997cefe7"} Jul 09 14:15:24.304600 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.304555 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vcf89" podStartSLOduration=1.9497718750000002 podStartE2EDuration="7.304539265s" totalImagesPullingTime="5.35476739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:24.303876991 +0000 UTC m=+44.908391697" watchObservedRunningTime="2026-07-09 14:15:24.304539265 +0000 UTC m=+44.909053964" Jul 09 14:15:24.380633 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:24.380599 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59478f8d8f-5h4rw"] Jul 09 14:15:25.296246 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.296011 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2lkd" event={"ID":"df2b62f9-61e7-44ed-8880-3772033e8e9c","Type":"ContainerStarted","Data":"6de35615873c4910731c00496909833c661ae84447bd6eba04f136891a67b1bb"} Jul 09 14:15:25.296882 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.296312 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:25.296882 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.296333 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2lkd" event={"ID":"df2b62f9-61e7-44ed-8880-3772033e8e9c","Type":"ContainerStarted","Data":"a6073054f498dc18d9b7c82e69e3909aa17547e79195bda9755f0084aef0a15f"} Jul 09 14:15:25.298639 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.298604 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" event={"ID":"591f3a7b-19a2-4ad1-ae4b-39b7e852811b","Type":"ContainerStarted","Data":"ad748625ed2d42d41eae6edfa88ee61946f87b3477395a2a8c29f0d53478ce47"} Jul 09 14:15:25.298777 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.298665 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" event={"ID":"591f3a7b-19a2-4ad1-ae4b-39b7e852811b","Type":"ContainerStarted","Data":"a9a7bf9a279ff1c1310b5b9cc0927fce6ed60c68275cdbf76085ada8a52d1aa1"} Jul 09 14:15:25.300287 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.300261 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd8dl" event={"ID":"e1aae7da-8e11-42fc-b740-ee4e35e26793","Type":"ContainerStarted","Data":"23b6601ad76e3b53f689adfd168082ebbc840b41384c12e4d1bafcccd53cbf65"} Jul 09 14:15:25.301498 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.301469 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59478f8d8f-5h4rw" event={"ID":"6995c392-15cf-4a9b-8837-622f3f360654","Type":"ContainerStarted","Data":"7e968216a219623a4487900e62bd7dd0e8f63af310340afdc6f390e2fd9a0ed6"} Jul 09 14:15:25.316041 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.315998 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c2lkd" podStartSLOduration=2.186264557 podStartE2EDuration="8.315980756s" totalImagesPullingTime="6.129716199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:25.314924851 +0000 UTC m=+45.919439551" watchObservedRunningTime="2026-07-09 14:15:25.315980756 +0000 UTC m=+45.920495457" Jul 09 14:15:25.632690 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.632662 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:25.701532 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.701503 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2"] Jul 09 14:15:25.714203 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.714146 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:25.715123 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.715102 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2"] Jul 09 14:15:25.715123 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.715122 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:25.717864 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.717459 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 14:15:25.717864 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.717717 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rqn55\"" Jul 09 14:15:25.803309 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.803235 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdnr4\" (UniqueName: \"kubernetes.io/projected/d0ec1ee5-583c-4438-943f-7dae1d2bc5b5-kube-api-access-bdnr4\") pod \"monitoring-plugin-5f68df7f58-wp8d2\" (UID: \"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:25.803510 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.803319 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0ec1ee5-583c-4438-943f-7dae1d2bc5b5-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-wp8d2\" (UID: \"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:25.886481 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.886386 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:25.904781 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.904729 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0ec1ee5-583c-4438-943f-7dae1d2bc5b5-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-wp8d2\" (UID: \"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:25.904927 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.904821 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdnr4\" (UniqueName: \"kubernetes.io/projected/d0ec1ee5-583c-4438-943f-7dae1d2bc5b5-kube-api-access-bdnr4\") pod \"monitoring-plugin-5f68df7f58-wp8d2\" (UID: \"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:25.909552 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.909528 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0ec1ee5-583c-4438-943f-7dae1d2bc5b5-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-wp8d2\" (UID: \"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:25.927822 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:25.927771 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdnr4\" (UniqueName: \"kubernetes.io/projected/d0ec1ee5-583c-4438-943f-7dae1d2bc5b5-kube-api-access-bdnr4\") pod \"monitoring-plugin-5f68df7f58-wp8d2\" (UID: \"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:26.027033 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:26.027002 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:26.194204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:26.193916 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2"] Jul 09 14:15:26.310333 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:26.310277 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd8dl" event={"ID":"e1aae7da-8e11-42fc-b740-ee4e35e26793","Type":"ContainerStarted","Data":"237c34720e5b6eff04577f30ed29ead3435e6679fcc940fbd270b9a9f948cff8"} Jul 09 14:15:26.344153 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:26.344117 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:26.445487 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:26.445363 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:26.541783 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:26.541748 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ec1ee5_583c_4438_943f_7dae1d2bc5b5.slice/crio-f56d1d362c8840d03cf0aea5535c066cbb64caf6d0ba3b0bab4287c0c098c74d WatchSource:0}: Error finding container f56d1d362c8840d03cf0aea5535c066cbb64caf6d0ba3b0bab4287c0c098c74d: Status 404 returned error can't find the container with id f56d1d362c8840d03cf0aea5535c066cbb64caf6d0ba3b0bab4287c0c098c74d Jul 09 14:15:27.130387 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.130353 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:27.133759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.133624 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:15:27.150642 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.150496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.152436 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.152392 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:15:27.154098 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.153965 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 14:15:27.154313 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.154282 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 14:15:27.154447 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.154383 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8q7f6ffddb2me\"" Jul 09 14:15:27.156051 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.155530 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 14:15:27.156051 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.155848 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 14:15:27.156211 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.156131 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 14:15:27.156775 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.156554 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 14:15:27.156775 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.156631 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 14:15:27.157978 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.156856 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 14:15:27.157978 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.157057 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 14:15:27.157978 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.157322 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 14:15:27.157978 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.157342 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-77j6f\"" Jul 09 14:15:27.157978 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.157748 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 14:15:27.157978 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.157761 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 14:15:27.158723 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.158004 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 14:15:27.158723 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.158175 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 14:15:27.159529 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.159367 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 14:15:27.217533 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217449 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6q6\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-kube-api-access-vb6q6\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.217533 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.217774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.217774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217641 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-config\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.217774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217698 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.217774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217728 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.217774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217769 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217831 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217881 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217914 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217937 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.217975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.218002 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.218024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.218054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.218084 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.218107 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.218135 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.218350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.218160 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.317113 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.317078 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" event={"ID":"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5","Type":"ContainerStarted","Data":"f56d1d362c8840d03cf0aea5535c066cbb64caf6d0ba3b0bab4287c0c098c74d"} Jul 09 14:15:27.318594 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318564 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318605 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318639 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318669 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318694 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318745 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318781 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6q6\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-kube-api-access-vb6q6\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318827 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318865 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-config\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318889 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318918 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.318968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.318962 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.319666 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.319034 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.319666 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.319058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.319666 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.319086 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.319666 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.319123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.319666 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.319149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.319666 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.319194 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.321373 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.320478 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.321373 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.320948 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.321373 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.320948 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.321373 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.321313 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.323611 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.323549 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.324337 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.324218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.324337 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.324231 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.325041 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.324601 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.325041 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.324665 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.325041 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.324701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.326524 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.325845 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.326524 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.326480 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-config\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.328228 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.328183 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.328318 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.328291 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.329927 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.329853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.330584 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.330070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.330584 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.330076 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.330584 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.330452 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.330584 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.330550 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6q6\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-kube-api-access-vb6q6\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.331545 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.331339 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:27.466471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:27.466373 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:30.074656 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.074626 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:15:30.137553 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.137479 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:30.182984 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:30.182956 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a41a48_4041_443e_8768_72db77f0cd28.slice/crio-7ca52e0fa2f84ef22b17fdc4621d3236a886b1c807f7da43e7285cecebeab9ff WatchSource:0}: Error finding container 7ca52e0fa2f84ef22b17fdc4621d3236a886b1c807f7da43e7285cecebeab9ff: Status 404 returned error can't find the container with id 7ca52e0fa2f84ef22b17fdc4621d3236a886b1c807f7da43e7285cecebeab9ff Jul 09 14:15:30.238132 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.238108 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:30.334812 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.334720 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" event={"ID":"fda6cf0f-675a-48a5-9b8d-b2e390b37031","Type":"ContainerStarted","Data":"a1364ee1dedc462d769927ff76cd781dd3ca6a997e9f5ef7a27b297e28475b69"} Jul 09 14:15:30.336347 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.336289 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerStarted","Data":"7ca52e0fa2f84ef22b17fdc4621d3236a886b1c807f7da43e7285cecebeab9ff"} Jul 09 14:15:30.338775 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.338587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:30.339515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.339353 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" event={"ID":"d0ec1ee5-583c-4438-943f-7dae1d2bc5b5","Type":"ContainerStarted","Data":"388be83c411f985627e5665577378a9c54f5ad0244781c6a55e248a96a18cc6d"} Jul 09 14:15:30.340020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.339939 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:30.342791 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.342764 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" event={"ID":"591f3a7b-19a2-4ad1-ae4b-39b7e852811b","Type":"ContainerStarted","Data":"3091655044ad35feeca04cd7ee5e2b72fbb8cee2943f061b500d97c38e5aafd7"} Jul 09 14:15:30.345538 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.345516 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" Jul 09 14:15:30.345678 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.345658 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pktfk" event={"ID":"9b50e9c2-964e-4a44-a882-7476b118808f","Type":"ContainerStarted","Data":"94be6a199e9f29a38905bc2323c5241bb02da72013e93082ddb659cd16919343"} Jul 09 14:15:30.348671 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.348640 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nd8dl" event={"ID":"e1aae7da-8e11-42fc-b740-ee4e35e26793","Type":"ContainerStarted","Data":"3090085bf911270bb4baef6c14bf501d5710376ba83f3a9fa0a794f41ca3fc2d"} Jul 09 14:15:30.352453 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.352414 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59478f8d8f-5h4rw" event={"ID":"6995c392-15cf-4a9b-8837-622f3f360654","Type":"ContainerStarted","Data":"393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a"} Jul 09 14:15:30.358454 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.358383 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wp8d2" podStartSLOduration=1.697475328 podStartE2EDuration="5.358367697s" totalImagesPullingTime="3.660892369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:30.357390401 +0000 UTC m=+50.961905102" watchObservedRunningTime="2026-07-09 14:15:30.358367697 +0000 UTC m=+50.962882625" Jul 09 14:15:30.383402 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.383344 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-sfmh7" podStartSLOduration=4.695953207 podStartE2EDuration="10.383314868s" totalImagesPullingTime="5.687361661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:30.382841903 +0000 UTC m=+50.987356640" watchObservedRunningTime="2026-07-09 14:15:30.383314868 +0000 UTC m=+50.987829569" Jul 09 14:15:30.421959 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.421606 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59478f8d8f-5h4rw" podStartSLOduration=1.674484208 podStartE2EDuration="7.421584259s" totalImagesPullingTime="5.747100051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:30.404600621 +0000 UTC m=+51.009115322" watchObservedRunningTime="2026-07-09 14:15:30.421584259 +0000 UTC m=+51.026098956" Jul 09 14:15:30.450595 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:30.449096 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nd8dl" podStartSLOduration=1.751738529 podStartE2EDuration="13.449040752s" totalImagesPullingTime="11.697302223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:15:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:30.446256488 +0000 UTC m=+51.050771189" watchObservedRunningTime="2026-07-09 14:15:30.449040752 +0000 UTC m=+51.053555449" Jul 09 14:15:31.361909 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:31.361862 2556 generic.go:356] "Generic (PLEG): container finished" podID="9b50e9c2-964e-4a44-a882-7476b118808f" containerID="94be6a199e9f29a38905bc2323c5241bb02da72013e93082ddb659cd16919343" exitCode=0 Jul 09 14:15:31.362382 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:31.361949 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pktfk" event={"ID":"9b50e9c2-964e-4a44-a882-7476b118808f","Type":"ContainerDied","Data":"94be6a199e9f29a38905bc2323c5241bb02da72013e93082ddb659cd16919343"} Jul 09 14:15:31.364316 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:31.364291 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" event={"ID":"fda6cf0f-675a-48a5-9b8d-b2e390b37031","Type":"ContainerStarted","Data":"a7c026e1c0b433944fb3ee70319a3e1456bef6c7901f5cf178d51cd65d14e4c5"} Jul 09 14:15:31.364445 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:31.364335 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" event={"ID":"fda6cf0f-675a-48a5-9b8d-b2e390b37031","Type":"ContainerStarted","Data":"114f9cbfea3c65f43a0e4ac85b7e4350a0b0026a508bacef77a190249748b866"} Jul 09 14:15:31.400719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:31.400679 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-f8htr" podStartSLOduration=5.476024432 podStartE2EDuration="11.400664405s" totalImagesPullingTime="5.924639973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:31.399808475 +0000 UTC m=+52.004323172" watchObservedRunningTime="2026-07-09 14:15:31.400664405 +0000 UTC m=+52.005179104" Jul 09 14:15:32.370452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:32.370405 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pktfk" event={"ID":"9b50e9c2-964e-4a44-a882-7476b118808f","Type":"ContainerStarted","Data":"4c2b9e70b25993a81d89de0e7442591f7f25eba90f1b881ad9a0c537bf6f0b64"} Jul 09 14:15:32.370904 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:32.370474 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pktfk" event={"ID":"9b50e9c2-964e-4a44-a882-7476b118808f","Type":"ContainerStarted","Data":"1c19003e2e58192322b1d98b0f0445727c8ad6fd4a525c8c5c3783812a9ae800"} Jul 09 14:15:32.393037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:32.392990 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pktfk" podStartSLOduration=6.127948303 podStartE2EDuration="12.392973131s" totalImagesPullingTime="6.166061514s" totalInitContainerRuntime="98.963314ms" isStatefulPod=true podCreationTimestamp="2026-07-09 14:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:32.392558086 +0000 UTC m=+52.997072785" watchObservedRunningTime="2026-07-09 14:15:32.392973131 +0000 UTC m=+52.997487831" Jul 09 14:15:32.772623 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:32.772537 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:33.378331 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:33.378234 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6a41a48-4041-443e-8768-72db77f0cd28" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" exitCode=0 Jul 09 14:15:33.378331 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:33.378296 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089"} Jul 09 14:15:34.236713 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:34.236671 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:34.236918 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:34.236737 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:34.242439 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:34.242392 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:34.385703 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:34.385676 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:15:34.626476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:34.626439 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:34.689193 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:34.689165 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:35.312493 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:35.312468 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c2lkd" Jul 09 14:15:35.746215 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:35.746128 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:37.395100 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:37.395065 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerStarted","Data":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} Jul 09 14:15:37.395498 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:37.395129 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerStarted","Data":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} Jul 09 14:15:37.596749 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:37.596719 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:37.696813 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:37.696736 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:40.407749 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:40.407713 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerStarted","Data":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} Jul 09 14:15:40.408105 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:40.407786 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerStarted","Data":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} Jul 09 14:15:40.408105 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:40.407812 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerStarted","Data":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} Jul 09 14:15:40.408105 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:40.407837 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerStarted","Data":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} Jul 09 14:15:40.436000 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:40.435954 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.615890228 podStartE2EDuration="13.435939099s" totalImagesPullingTime="8.803675907s" totalInitContainerRuntime="16.372964ms" isStatefulPod=false podCreationTimestamp="2026-07-09 14:15:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:40.435279646 +0000 UTC m=+61.039794337" watchObservedRunningTime="2026-07-09 14:15:40.435939099 +0000 UTC m=+61.040453798" Jul 09 14:15:41.314889 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:41.314854 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:41.469729 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:41.469706 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:41.488201 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:41.488180 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:42.073907 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:42.073873 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:42.172536 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:42.172490 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:42.232836 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:42.232810 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hhjnx" Jul 09 14:15:42.365341 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:42.365251 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:42.415513 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:42.415485 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:42.426187 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:42.426161 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:42.430803 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:42.430780 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:15:43.368999 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:43.368964 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:43.470136 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:43.470105 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:44.685802 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.685754 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:44.685802 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.685803 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:44.688791 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.688773 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 14:15:44.689456 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.689440 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 14:15:44.698581 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.698555 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bbda19-f3b2-4d1d-8aaa-1c1647b55efa-metrics-certs\") pod \"network-metrics-daemon-s7f2d\" (UID: \"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa\") " pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:44.698581 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.698565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e988dcf9-c3bd-4bc1-95d0-d5217096ec97-original-pull-secret\") pod \"global-pull-secret-syncer-tmpjn\" (UID: \"e988dcf9-c3bd-4bc1-95d0-d5217096ec97\") " pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:44.786825 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.786785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:44.789600 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.789584 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 14:15:44.799719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.799704 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 14:15:44.810614 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.810594 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcsk\" (UniqueName: \"kubernetes.io/projected/5545addf-2be8-42e2-a1f6-603d81bed70f-kube-api-access-gvcsk\") pod \"network-check-target-b5ssg\" (UID: \"5545addf-2be8-42e2-a1f6-603d81bed70f\") " pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:44.890864 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.890832 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cs7kw\"" Jul 09 14:15:44.897892 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.897873 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s7f2d" Jul 09 14:15:44.901586 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.901564 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wzd4v\"" Jul 09 14:15:44.908392 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.908365 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmpjn" Jul 09 14:15:44.909450 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.909414 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:44.924787 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:44.924758 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:45.024410 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.024364 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:45.070569 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.070450 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s7f2d"] Jul 09 14:15:45.073610 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:45.073574 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bbda19_f3b2_4d1d_8aaa_1c1647b55efa.slice/crio-536122a3a5c9bef93330bfe049db8b55981fb586d49332b43d4ee3ec2af16e94 WatchSource:0}: Error finding container 536122a3a5c9bef93330bfe049db8b55981fb586d49332b43d4ee3ec2af16e94: Status 404 returned error can't find the container with id 536122a3a5c9bef93330bfe049db8b55981fb586d49332b43d4ee3ec2af16e94 Jul 09 14:15:45.081673 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.081601 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5ssg"] Jul 09 14:15:45.083837 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:45.083815 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5545addf_2be8_42e2_a1f6_603d81bed70f.slice/crio-aad37a5dd6a4a4e9a66eb5bdc7f32a50d2cc35fd11a1ba6b218214b2df2df232 WatchSource:0}: Error finding container aad37a5dd6a4a4e9a66eb5bdc7f32a50d2cc35fd11a1ba6b218214b2df2df232: Status 404 returned error can't find the container with id aad37a5dd6a4a4e9a66eb5bdc7f32a50d2cc35fd11a1ba6b218214b2df2df232 Jul 09 14:15:45.101185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.101159 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tmpjn"] Jul 09 14:15:45.104016 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:15:45.103994 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode988dcf9_c3bd_4bc1_95d0_d5217096ec97.slice/crio-2522d9d63c7b510664f080db0e868f4da04871abb41e2298fa28b368566d9aa7 WatchSource:0}: Error finding container 2522d9d63c7b510664f080db0e868f4da04871abb41e2298fa28b368566d9aa7: Status 404 returned error can't find the container with id 2522d9d63c7b510664f080db0e868f4da04871abb41e2298fa28b368566d9aa7 Jul 09 14:15:45.241272 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.241166 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:45.367279 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.367235 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:45.426727 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.426690 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7f2d" event={"ID":"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa","Type":"ContainerStarted","Data":"536122a3a5c9bef93330bfe049db8b55981fb586d49332b43d4ee3ec2af16e94"} Jul 09 14:15:45.427743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.427722 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tmpjn" event={"ID":"e988dcf9-c3bd-4bc1-95d0-d5217096ec97","Type":"ContainerStarted","Data":"2522d9d63c7b510664f080db0e868f4da04871abb41e2298fa28b368566d9aa7"} Jul 09 14:15:45.429252 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.429228 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5ssg" event={"ID":"5545addf-2be8-42e2-a1f6-603d81bed70f","Type":"ContainerStarted","Data":"aad37a5dd6a4a4e9a66eb5bdc7f32a50d2cc35fd11a1ba6b218214b2df2df232"} Jul 09 14:15:45.766874 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.766818 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:45.826065 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:45.826034 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:47.443343 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:47.443299 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7f2d" event={"ID":"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa","Type":"ContainerStarted","Data":"3c5e06928bb1a5ec523f893ab095ee5d6c185ff97486b8d0a00112d5d66c4f67"} Jul 09 14:15:47.443734 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:47.443376 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s7f2d" event={"ID":"03bbda19-f3b2-4d1d-8aaa-1c1647b55efa","Type":"ContainerStarted","Data":"48d14c9350dbe053302254dda48dcf8f0e313296780a133d4c7c8f65dbcdb7df"} Jul 09 14:15:47.460656 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:47.460560 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s7f2d" podStartSLOduration=65.925604461 podStartE2EDuration="1m7.460538104s" totalImagesPullingTime="1.534933643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:47.46038764 +0000 UTC m=+68.064902357" watchObservedRunningTime="2026-07-09 14:15:47.460538104 +0000 UTC m=+68.065052806" Jul 09 14:15:47.559545 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:47.559417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:47.636947 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:47.636910 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:48.478996 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:48.478940 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:48.548105 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:48.548069 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:51.434569 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.434529 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59478f8d8f-5h4rw"] Jul 09 14:15:51.461708 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.461673 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tmpjn" event={"ID":"e988dcf9-c3bd-4bc1-95d0-d5217096ec97","Type":"ContainerStarted","Data":"f9084505f947378aa4c1d7880cb9a88124e0cba2774b5b0d06d727e9ee1d2684"} Jul 09 14:15:51.463126 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.463105 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5ssg" event={"ID":"5545addf-2be8-42e2-a1f6-603d81bed70f","Type":"ContainerStarted","Data":"1732d23e28c658a30ca2ef0959176666b9f9c25556c2cfee940f687aaaac2671"} Jul 09 14:15:51.463243 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.463224 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:15:51.477253 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.477212 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tmpjn" podStartSLOduration=65.558184102 podStartE2EDuration="1m11.477199423s" totalImagesPullingTime="5.919015321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:51.475990807 +0000 UTC m=+72.080505507" watchObservedRunningTime="2026-07-09 14:15:51.477199423 +0000 UTC m=+72.081714123" Jul 09 14:15:51.491632 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.491584 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b5ssg" podStartSLOduration=65.555295156 podStartE2EDuration="1m11.491567019s" totalImagesPullingTime="5.936271863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:15:51.490832315 +0000 UTC m=+72.095347014" watchObservedRunningTime="2026-07-09 14:15:51.491567019 +0000 UTC m=+72.096081720" Jul 09 14:15:51.542710 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.542684 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:51.644098 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.644065 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:51.712263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:51.712191 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:52.451597 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:52.451561 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:15:52.512291 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:15:52.512247 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:00.225814 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:00.225780 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:00.747176 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:00.747138 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:06.341988 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:06.341948 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:06.443241 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:06.443209 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:06.691979 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:06.691902 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:06.793038 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:06.793010 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:06.893065 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:06.893034 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:07.436854 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:07.436824 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:07.790804 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:07.790724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:11.896189 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:11.896154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:12.476015 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:12.475980 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:12.562080 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:12.562046 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:13.700504 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:13.700478 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:13.801301 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:13.801272 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:14.864673 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:14.864643 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:16:14.865113 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:14.864947 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="prometheus" containerID="cri-o://40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" gracePeriod=600 Jul 09 14:16:14.865113 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:14.865004 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="thanos-sidecar" containerID="cri-o://eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" gracePeriod=600 Jul 09 14:16:14.865113 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:14.865019 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-web" containerID="cri-o://8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" gracePeriod=600 Jul 09 14:16:14.865113 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:14.865039 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-thanos" containerID="cri-o://56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" gracePeriod=600 Jul 09 14:16:14.865113 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:14.865073 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="config-reloader" containerID="cri-o://d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" gracePeriod=600 Jul 09 14:16:14.865326 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:14.865005 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy" containerID="cri-o://d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" gracePeriod=600 Jul 09 14:16:15.102897 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.102874 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.127535 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127483 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-tls\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127535 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127514 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-0\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127702 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127539 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-kube-rbac-proxy\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127702 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127555 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-metrics-client-ca\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127702 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127572 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb6q6\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-kube-api-access-vb6q6\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127702 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127676 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-2\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127721 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-kubelet-serving-ca-bundle\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127760 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127798 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-metrics-client-certs\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127833 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-db\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127858 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-grpc-tls\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.127899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127892 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-config-out\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127934 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-config\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127971 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-thanos-prometheus-http-client-file\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128014 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128045 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-web-config\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128069 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-trusted-ca-bundle\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128112 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-1\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128152 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-serving-certs-ca-bundle\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128180 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-tls-assets\") pod \"c6a41a48-4041-443e-8768-72db77f0cd28\" (UID: \"c6a41a48-4041-443e-8768-72db77f0cd28\") " Jul 09 14:16:15.128644 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.127965 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:15.128644 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128472 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-metrics-client-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.129246 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.129144 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:15.129893 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.129716 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:16:15.129893 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128294 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:15.129893 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.128339 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:15.130108 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.129983 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:15.130347 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.130319 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:15.130588 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.130539 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:15.130588 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.130576 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-kube-api-access-vb6q6" (OuterVolumeSpecName: "kube-api-access-vb6q6") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "kube-api-access-vb6q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:16:15.130790 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.130647 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:16:15.130790 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.130696 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.131485 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.131438 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.132063 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.132033 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.132181 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.132148 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.132308 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.132245 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.132644 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.132620 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.132712 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.132664 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-config-out" (OuterVolumeSpecName: "config-out") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:16:15.132712 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.132673 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-config" (OuterVolumeSpecName: "config") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.132822 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.132804 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.144628 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.144599 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-web-config" (OuterVolumeSpecName: "web-config") pod "c6a41a48-4041-443e-8768-72db77f0cd28" (UID: "c6a41a48-4041-443e-8768-72db77f0cd28"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:15.229136 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229113 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-tls-assets\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229136 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229133 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229144 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229152 2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-kube-rbac-proxy\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229161 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6q6\" (UniqueName: \"kubernetes.io/projected/c6a41a48-4041-443e-8768-72db77f0cd28-kube-api-access-vb6q6\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229169 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229178 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229189 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229198 2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-metrics-client-certs\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229205 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-db\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229215 2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-grpc-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229223 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a41a48-4041-443e-8768-72db77f0cd28-config-out\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229230 2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-config\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229239 2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229247 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229274 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a41a48-4041-443e-8768-72db77f0cd28-web-config\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229283 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229289 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229292 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.229750 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.229300 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a41a48-4041-443e-8768-72db77f0cd28-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:15.260241 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.260212 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:15.361274 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.361153 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:15.547054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.546963 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6a41a48-4041-443e-8768-72db77f0cd28" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" exitCode=0 Jul 09 14:16:15.547054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.546990 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6a41a48-4041-443e-8768-72db77f0cd28" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" exitCode=0 Jul 09 14:16:15.547054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.546998 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6a41a48-4041-443e-8768-72db77f0cd28" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" exitCode=0 Jul 09 14:16:15.547054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547006 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6a41a48-4041-443e-8768-72db77f0cd28" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" exitCode=0 Jul 09 14:16:15.547054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547013 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6a41a48-4041-443e-8768-72db77f0cd28" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" exitCode=0 Jul 09 14:16:15.547054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547019 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6a41a48-4041-443e-8768-72db77f0cd28" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" exitCode=0 Jul 09 14:16:15.547054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547034 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547051 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547169 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547195 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547211 2556 scope.go:117] "RemoveContainer" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547221 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547248 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} Jul 09 14:16:15.547452 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.547272 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6a41a48-4041-443e-8768-72db77f0cd28","Type":"ContainerDied","Data":"7ca52e0fa2f84ef22b17fdc4621d3236a886b1c807f7da43e7285cecebeab9ff"} Jul 09 14:16:15.556894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.556875 2556 scope.go:117] "RemoveContainer" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.563920 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.563899 2556 scope.go:117] "RemoveContainer" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.572626 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.572088 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:16:15.577612 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.577595 2556 scope.go:117] "RemoveContainer" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.580211 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.580190 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:16:15.586950 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.586937 2556 scope.go:117] "RemoveContainer" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.593388 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.593372 2556 scope.go:117] "RemoveContainer" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.600180 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.600150 2556 scope.go:117] "RemoveContainer" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.606349 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.606327 2556 scope.go:117] "RemoveContainer" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.606630 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:15.606612 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": container with ID starting with 56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b not found: ID does not exist" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.606682 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.606637 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} err="failed to get container status \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": rpc error: code = NotFound desc = could not find container \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": container with ID starting with 56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b not found: ID does not exist" Jul 09 14:16:15.606682 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.606667 2556 scope.go:117] "RemoveContainer" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.606896 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:15.606878 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": container with ID starting with d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47 not found: ID does not exist" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.606960 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.606908 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} err="failed to get container status \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": rpc error: code = NotFound desc = could not find container \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": container with ID starting with d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47 not found: ID does not exist" Jul 09 14:16:15.606960 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.606930 2556 scope.go:117] "RemoveContainer" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.607152 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:15.607135 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": container with ID starting with 8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1 not found: ID does not exist" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.607193 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.607156 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} err="failed to get container status \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": rpc error: code = NotFound desc = could not find container \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": container with ID starting with 8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1 not found: ID does not exist" Jul 09 14:16:15.607193 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.607172 2556 scope.go:117] "RemoveContainer" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.607371 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:15.607355 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": container with ID starting with eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58 not found: ID does not exist" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.607444 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.607378 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} err="failed to get container status \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": rpc error: code = NotFound desc = could not find container \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": container with ID starting with eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58 not found: ID does not exist" Jul 09 14:16:15.607444 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.607400 2556 scope.go:117] "RemoveContainer" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.607678 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:15.607623 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": container with ID starting with d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf not found: ID does not exist" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.607795 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.607672 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} err="failed to get container status \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": rpc error: code = NotFound desc = could not find container \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": container with ID starting with d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf not found: ID does not exist" Jul 09 14:16:15.607795 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.607691 2556 scope.go:117] "RemoveContainer" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.608006 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:15.607982 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": container with ID starting with 40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c not found: ID does not exist" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.608075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.608014 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} err="failed to get container status \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": rpc error: code = NotFound desc = could not find container \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": container with ID starting with 40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c not found: ID does not exist" Jul 09 14:16:15.608075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.608034 2556 scope.go:117] "RemoveContainer" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.608547 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:15.608529 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": container with ID starting with a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089 not found: ID does not exist" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.608677 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.608555 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089"} err="failed to get container status \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": rpc error: code = NotFound desc = could not find container \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": container with ID starting with a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089 not found: ID does not exist" Jul 09 14:16:15.608677 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.608575 2556 scope.go:117] "RemoveContainer" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.609151 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609131 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} err="failed to get container status \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": rpc error: code = NotFound desc = could not find container \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": container with ID starting with 56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b not found: ID does not exist" Jul 09 14:16:15.609151 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609151 2556 scope.go:117] "RemoveContainer" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.609400 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609380 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} err="failed to get container status \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": rpc error: code = NotFound desc = could not find container \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": container with ID starting with d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47 not found: ID does not exist" Jul 09 14:16:15.609472 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609401 2556 scope.go:117] "RemoveContainer" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.609624 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609607 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:16:15.609663 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609621 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} err="failed to get container status \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": rpc error: code = NotFound desc = could not find container \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": container with ID starting with 8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1 not found: ID does not exist" Jul 09 14:16:15.609663 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609635 2556 scope.go:117] "RemoveContainer" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.609884 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609865 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} err="failed to get container status \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": rpc error: code = NotFound desc = could not find container \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": container with ID starting with eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58 not found: ID does not exist" Jul 09 14:16:15.609930 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609886 2556 scope.go:117] "RemoveContainer" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.609930 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609901 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="init-config-reloader" Jul 09 14:16:15.609930 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609916 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="init-config-reloader" Jul 09 14:16:15.609930 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609924 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="thanos-sidecar" Jul 09 14:16:15.609930 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609930 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="thanos-sidecar" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609941 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-web" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609950 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-web" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609958 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609967 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609983 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-thanos" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.609991 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-thanos" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610002 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="config-reloader" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610010 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="config-reloader" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610024 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="prometheus" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610033 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="prometheus" Jul 09 14:16:15.610106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610094 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} err="failed to get container status \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": rpc error: code = NotFound desc = could not find container \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": container with ID starting with d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf not found: ID does not exist" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610113 2556 scope.go:117] "RemoveContainer" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610100 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="prometheus" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610154 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="config-reloader" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610161 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610170 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="thanos-sidecar" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610175 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-web" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610183 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" containerName="kube-rbac-proxy-thanos" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610362 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} err="failed to get container status \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": rpc error: code = NotFound desc = could not find container \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": container with ID starting with 40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c not found: ID does not exist" Jul 09 14:16:15.610476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610381 2556 scope.go:117] "RemoveContainer" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.610755 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610652 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089"} err="failed to get container status \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": rpc error: code = NotFound desc = could not find container \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": container with ID starting with a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089 not found: ID does not exist" Jul 09 14:16:15.610755 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610669 2556 scope.go:117] "RemoveContainer" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.610906 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610885 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} err="failed to get container status \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": rpc error: code = NotFound desc = could not find container \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": container with ID starting with 56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b not found: ID does not exist" Jul 09 14:16:15.610947 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.610908 2556 scope.go:117] "RemoveContainer" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.611151 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611132 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} err="failed to get container status \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": rpc error: code = NotFound desc = could not find container \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": container with ID starting with d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47 not found: ID does not exist" Jul 09 14:16:15.611209 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611151 2556 scope.go:117] "RemoveContainer" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.611376 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611359 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} err="failed to get container status \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": rpc error: code = NotFound desc = could not find container \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": container with ID starting with 8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1 not found: ID does not exist" Jul 09 14:16:15.611413 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611377 2556 scope.go:117] "RemoveContainer" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.611599 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611579 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} err="failed to get container status \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": rpc error: code = NotFound desc = could not find container \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": container with ID starting with eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58 not found: ID does not exist" Jul 09 14:16:15.611652 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611599 2556 scope.go:117] "RemoveContainer" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.611780 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611763 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} err="failed to get container status \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": rpc error: code = NotFound desc = could not find container \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": container with ID starting with d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf not found: ID does not exist" Jul 09 14:16:15.611821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611781 2556 scope.go:117] "RemoveContainer" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.611983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611961 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} err="failed to get container status \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": rpc error: code = NotFound desc = could not find container \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": container with ID starting with 40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c not found: ID does not exist" Jul 09 14:16:15.611983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.611980 2556 scope.go:117] "RemoveContainer" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.612162 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612145 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089"} err="failed to get container status \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": rpc error: code = NotFound desc = could not find container \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": container with ID starting with a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089 not found: ID does not exist" Jul 09 14:16:15.612209 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612162 2556 scope.go:117] "RemoveContainer" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.612441 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612401 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} err="failed to get container status \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": rpc error: code = NotFound desc = could not find container \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": container with ID starting with 56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b not found: ID does not exist" Jul 09 14:16:15.612441 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612437 2556 scope.go:117] "RemoveContainer" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.612691 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612674 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} err="failed to get container status \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": rpc error: code = NotFound desc = could not find container \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": container with ID starting with d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47 not found: ID does not exist" Jul 09 14:16:15.612731 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612691 2556 scope.go:117] "RemoveContainer" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.612910 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612894 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} err="failed to get container status \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": rpc error: code = NotFound desc = could not find container \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": container with ID starting with 8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1 not found: ID does not exist" Jul 09 14:16:15.612960 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.612910 2556 scope.go:117] "RemoveContainer" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.613115 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613101 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} err="failed to get container status \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": rpc error: code = NotFound desc = could not find container \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": container with ID starting with eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58 not found: ID does not exist" Jul 09 14:16:15.613115 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613114 2556 scope.go:117] "RemoveContainer" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.613325 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613307 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} err="failed to get container status \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": rpc error: code = NotFound desc = could not find container \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": container with ID starting with d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf not found: ID does not exist" Jul 09 14:16:15.613376 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613326 2556 scope.go:117] "RemoveContainer" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.613551 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613531 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} err="failed to get container status \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": rpc error: code = NotFound desc = could not find container \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": container with ID starting with 40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c not found: ID does not exist" Jul 09 14:16:15.613617 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613550 2556 scope.go:117] "RemoveContainer" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.613778 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613760 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089"} err="failed to get container status \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": rpc error: code = NotFound desc = could not find container \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": container with ID starting with a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089 not found: ID does not exist" Jul 09 14:16:15.613815 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613783 2556 scope.go:117] "RemoveContainer" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.613886 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.613873 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.614294 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614050 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} err="failed to get container status \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": rpc error: code = NotFound desc = could not find container \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": container with ID starting with 56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b not found: ID does not exist" Jul 09 14:16:15.614294 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614077 2556 scope.go:117] "RemoveContainer" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.614516 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614496 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} err="failed to get container status \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": rpc error: code = NotFound desc = could not find container \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": container with ID starting with d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47 not found: ID does not exist" Jul 09 14:16:15.614516 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614516 2556 scope.go:117] "RemoveContainer" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.614742 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614712 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} err="failed to get container status \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": rpc error: code = NotFound desc = could not find container \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": container with ID starting with 8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1 not found: ID does not exist" Jul 09 14:16:15.614742 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614734 2556 scope.go:117] "RemoveContainer" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.614963 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614942 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} err="failed to get container status \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": rpc error: code = NotFound desc = could not find container \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": container with ID starting with eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58 not found: ID does not exist" Jul 09 14:16:15.615039 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.614965 2556 scope.go:117] "RemoveContainer" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.615242 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615224 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} err="failed to get container status \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": rpc error: code = NotFound desc = could not find container \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": container with ID starting with d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf not found: ID does not exist" Jul 09 14:16:15.615311 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615243 2556 scope.go:117] "RemoveContainer" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.615489 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615472 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} err="failed to get container status \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": rpc error: code = NotFound desc = could not find container \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": container with ID starting with 40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c not found: ID does not exist" Jul 09 14:16:15.615554 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615490 2556 scope.go:117] "RemoveContainer" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.615713 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615695 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089"} err="failed to get container status \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": rpc error: code = NotFound desc = could not find container \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": container with ID starting with a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089 not found: ID does not exist" Jul 09 14:16:15.615770 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615715 2556 scope.go:117] "RemoveContainer" containerID="56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b" Jul 09 14:16:15.615968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615948 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b"} err="failed to get container status \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": rpc error: code = NotFound desc = could not find container \"56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b\": container with ID starting with 56ccbc8e7cf30383abbe70b1fa3611a14caad525e85d2f8ae5158cb24fea708b not found: ID does not exist" Jul 09 14:16:15.615968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.615967 2556 scope.go:117] "RemoveContainer" containerID="d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47" Jul 09 14:16:15.616176 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616159 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47"} err="failed to get container status \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": rpc error: code = NotFound desc = could not find container \"d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47\": container with ID starting with d2af14f74610fbd3157a382e4707f9f82045fb502028a6bf2f95e2c427efcb47 not found: ID does not exist" Jul 09 14:16:15.616245 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616176 2556 scope.go:117] "RemoveContainer" containerID="8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1" Jul 09 14:16:15.616395 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616375 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1"} err="failed to get container status \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": rpc error: code = NotFound desc = could not find container \"8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1\": container with ID starting with 8e0b31705904cc31e61fbb70c60b2f90d39aaf7f934e0e5b83969da31fd195f1 not found: ID does not exist" Jul 09 14:16:15.616460 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616396 2556 scope.go:117] "RemoveContainer" containerID="eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58" Jul 09 14:16:15.616563 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616545 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 14:16:15.616646 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616627 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58"} err="failed to get container status \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": rpc error: code = NotFound desc = could not find container \"eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58\": container with ID starting with eca7f18b418bfbd1e715647dc981e661a8dd5c9560aad894c0a2c25e279a5d58 not found: ID does not exist" Jul 09 14:16:15.616704 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616649 2556 scope.go:117] "RemoveContainer" containerID="d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf" Jul 09 14:16:15.616704 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616695 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 14:16:15.616805 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616765 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 14:16:15.616805 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616779 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-77j6f\"" Jul 09 14:16:15.616926 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616884 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf"} err="failed to get container status \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": rpc error: code = NotFound desc = could not find container \"d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf\": container with ID starting with d3f336a0cfb448d9c82915c4df941bb88827a30c46156eee7b8b35cf1554b1bf not found: ID does not exist" Jul 09 14:16:15.616981 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.616928 2556 scope.go:117] "RemoveContainer" containerID="40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c" Jul 09 14:16:15.617103 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617083 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 14:16:15.617170 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617112 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 14:16:15.617170 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617150 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 14:16:15.617259 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617200 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 14:16:15.617259 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617205 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c"} err="failed to get container status \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": rpc error: code = NotFound desc = could not find container \"40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c\": container with ID starting with 40fec4f37d360b78461ea47d8bde7218fac0bd6a7cb3b557f9a40a6debe8576c not found: ID does not exist" Jul 09 14:16:15.617259 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617223 2556 scope.go:117] "RemoveContainer" containerID="a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089" Jul 09 14:16:15.617386 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617152 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8q7f6ffddb2me\"" Jul 09 14:16:15.617386 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617308 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 14:16:15.617524 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617484 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089"} err="failed to get container status \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": rpc error: code = NotFound desc = could not find container \"a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089\": container with ID starting with a4ed9a30b39e5117cb1ac8e6e6e834ac9d323cc4e34dd9a2d9e5d5997bf97089 not found: ID does not exist" Jul 09 14:16:15.617574 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617418 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 14:16:15.617618 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617587 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 14:16:15.617815 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617800 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 14:16:15.617873 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.617860 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 14:16:15.618141 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.618126 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 14:16:15.621960 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.621941 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 14:16:15.624226 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.624209 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633130 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633743 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633784 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633816 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjz5\" (UniqueName: \"kubernetes.io/projected/eb62817c-e209-49c6-9b43-b4290add5f80-kube-api-access-hvjz5\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633848 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633922 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.633963 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634002 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634025 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634085 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634125 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-config\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634149 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634198 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634226 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb62817c-e209-49c6-9b43-b4290add5f80-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634255 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634284 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634316 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb62817c-e209-49c6-9b43-b4290add5f80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634351 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.635699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.634380 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735031 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735005 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-config\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735031 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735032 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735070 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb62817c-e209-49c6-9b43-b4290add5f80-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735099 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb62817c-e209-49c6-9b43-b4290add5f80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735154 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735175 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735225 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735248 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjz5\" (UniqueName: \"kubernetes.io/projected/eb62817c-e209-49c6-9b43-b4290add5f80-kube-api-access-hvjz5\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735276 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735303 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735333 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735368 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735406 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735478 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.735589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735505 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.736057 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.736057 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.735880 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.736057 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.736034 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.736609 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.736585 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.737085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.737063 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.737782 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.737750 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb62817c-e209-49c6-9b43-b4290add5f80-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.738165 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.738045 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb62817c-e209-49c6-9b43-b4290add5f80-config-out\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.738165 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.738145 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.739215 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.738520 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.739215 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.738688 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-config\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.739215 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.739183 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.739411 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.739216 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.739952 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.739917 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.740044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.739994 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb62817c-e209-49c6-9b43-b4290add5f80-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.740316 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.740289 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.740403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.740347 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.740593 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.740576 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.740793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.740770 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.740891 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.740877 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb62817c-e209-49c6-9b43-b4290add5f80-web-config\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.748407 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.748388 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjz5\" (UniqueName: \"kubernetes.io/projected/eb62817c-e209-49c6-9b43-b4290add5f80-kube-api-access-hvjz5\") pod \"prometheus-k8s-0\" (UID: \"eb62817c-e209-49c6-9b43-b4290add5f80\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:15.923950 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:15.923923 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:16.063809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.063770 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 14:16:16.066460 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:16:16.066413 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb62817c_e209_49c6_9b43_b4290add5f80.slice/crio-781eba1df115671cb5a5fb530abaedcc40f33573a7ae0fd27780a202d32126c4 WatchSource:0}: Error finding container 781eba1df115671cb5a5fb530abaedcc40f33573a7ae0fd27780a202d32126c4: Status 404 returned error can't find the container with id 781eba1df115671cb5a5fb530abaedcc40f33573a7ae0fd27780a202d32126c4 Jul 09 14:16:16.075565 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.075473 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a41a48-4041-443e-8768-72db77f0cd28" path="/var/lib/kubelet/pods/c6a41a48-4041-443e-8768-72db77f0cd28/volumes" Jul 09 14:16:16.454556 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.454473 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59478f8d8f-5h4rw" podUID="6995c392-15cf-4a9b-8837-622f3f360654" containerName="console" containerID="cri-o://393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a" gracePeriod=15 Jul 09 14:16:16.552758 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.552729 2556 generic.go:356] "Generic (PLEG): container finished" podID="eb62817c-e209-49c6-9b43-b4290add5f80" containerID="a5767f1e3abc23e2a910f7c47bb716e785166b796295611f64f82fe8349b02b7" exitCode=0 Jul 09 14:16:16.552928 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.552825 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerDied","Data":"a5767f1e3abc23e2a910f7c47bb716e785166b796295611f64f82fe8349b02b7"} Jul 09 14:16:16.552928 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.552911 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerStarted","Data":"781eba1df115671cb5a5fb530abaedcc40f33573a7ae0fd27780a202d32126c4"} Jul 09 14:16:16.693778 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.693523 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:16:16.741913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.741636 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-oauth-serving-cert\") pod \"6995c392-15cf-4a9b-8837-622f3f360654\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " Jul 09 14:16:16.741913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.741696 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfrx9\" (UniqueName: \"kubernetes.io/projected/6995c392-15cf-4a9b-8837-622f3f360654-kube-api-access-dfrx9\") pod \"6995c392-15cf-4a9b-8837-622f3f360654\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " Jul 09 14:16:16.741913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.741726 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-serving-cert\") pod \"6995c392-15cf-4a9b-8837-622f3f360654\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " Jul 09 14:16:16.741913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.741786 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-console-config\") pod \"6995c392-15cf-4a9b-8837-622f3f360654\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " Jul 09 14:16:16.741913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.741820 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-oauth-config\") pod \"6995c392-15cf-4a9b-8837-622f3f360654\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " Jul 09 14:16:16.741913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.741863 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-service-ca\") pod \"6995c392-15cf-4a9b-8837-622f3f360654\" (UID: \"6995c392-15cf-4a9b-8837-622f3f360654\") " Jul 09 14:16:16.742309 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.742286 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-service-ca" (OuterVolumeSpecName: "service-ca") pod "6995c392-15cf-4a9b-8837-622f3f360654" (UID: "6995c392-15cf-4a9b-8837-622f3f360654"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:16.742770 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.742516 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-console-config" (OuterVolumeSpecName: "console-config") pod "6995c392-15cf-4a9b-8837-622f3f360654" (UID: "6995c392-15cf-4a9b-8837-622f3f360654"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:16.743770 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.743155 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6995c392-15cf-4a9b-8837-622f3f360654" (UID: "6995c392-15cf-4a9b-8837-622f3f360654"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:16:16.746191 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.745962 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6995c392-15cf-4a9b-8837-622f3f360654-kube-api-access-dfrx9" (OuterVolumeSpecName: "kube-api-access-dfrx9") pod "6995c392-15cf-4a9b-8837-622f3f360654" (UID: "6995c392-15cf-4a9b-8837-622f3f360654"). InnerVolumeSpecName "kube-api-access-dfrx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:16:16.746719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.746681 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6995c392-15cf-4a9b-8837-622f3f360654" (UID: "6995c392-15cf-4a9b-8837-622f3f360654"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:16.747881 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.747845 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6995c392-15cf-4a9b-8837-622f3f360654" (UID: "6995c392-15cf-4a9b-8837-622f3f360654"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:16:16.844246 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.844212 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-oauth-config\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:16.844402 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.844251 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-service-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:16.844402 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.844268 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-oauth-serving-cert\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:16.844402 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.844289 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfrx9\" (UniqueName: \"kubernetes.io/projected/6995c392-15cf-4a9b-8837-622f3f360654-kube-api-access-dfrx9\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:16.844402 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.844303 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6995c392-15cf-4a9b-8837-622f3f360654-console-serving-cert\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:16.844402 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:16.844318 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6995c392-15cf-4a9b-8837-622f3f360654-console-config\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:16:17.559687 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.559652 2556 generic.go:356] "Generic (PLEG): container finished" podID="6995c392-15cf-4a9b-8837-622f3f360654" containerID="393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a" exitCode=0 Jul 09 14:16:17.560085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.559733 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59478f8d8f-5h4rw" Jul 09 14:16:17.560085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.559734 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59478f8d8f-5h4rw" event={"ID":"6995c392-15cf-4a9b-8837-622f3f360654","Type":"ContainerDied","Data":"393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a"} Jul 09 14:16:17.560085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.559799 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59478f8d8f-5h4rw" event={"ID":"6995c392-15cf-4a9b-8837-622f3f360654","Type":"ContainerDied","Data":"7e968216a219623a4487900e62bd7dd0e8f63af310340afdc6f390e2fd9a0ed6"} Jul 09 14:16:17.560085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.559830 2556 scope.go:117] "RemoveContainer" containerID="393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a" Jul 09 14:16:17.564618 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.564584 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerStarted","Data":"3358502ebd64c8efb6ef0f93af879360ea370bd9895759569f492f952a0cf215"} Jul 09 14:16:17.564751 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.564667 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerStarted","Data":"5485ebe5d12413a7848470af4e7c5ce63e0e4c10543e57dc5e78e6772015156a"} Jul 09 14:16:17.564751 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.564709 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerStarted","Data":"7b342bc189a38802c259deaf667dd8ea697d42d2cc5474562aadca95b0801d9a"} Jul 09 14:16:17.564879 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.564751 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerStarted","Data":"51df11b2003a0a6ebab7c02c6bc458aaf16843da315a24ba4eac9336066929d2"} Jul 09 14:16:17.564879 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.564791 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerStarted","Data":"62dec2050732d2ba4ed7ab1cfa3d1eb5e1ff16134258b00a84ccd84235def439"} Jul 09 14:16:17.564879 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.564832 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eb62817c-e209-49c6-9b43-b4290add5f80","Type":"ContainerStarted","Data":"609f7e5275a0ef4943d6c64a006db08f5685cbb9d9e37999c17b7401ab069689"} Jul 09 14:16:17.569629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.569571 2556 scope.go:117] "RemoveContainer" containerID="393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a" Jul 09 14:16:17.569841 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:16:17.569819 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a\": container with ID starting with 393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a not found: ID does not exist" containerID="393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a" Jul 09 14:16:17.569913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.569851 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a"} err="failed to get container status \"393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a\": rpc error: code = NotFound desc = could not find container \"393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a\": container with ID starting with 393aa8e475b9afab5f5c44b5239078f923c53059fdae95511537dc95c4b2682a not found: ID does not exist" Jul 09 14:16:17.595895 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.595824 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.576298928 podStartE2EDuration="2.595812234s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.513306ms" isStatefulPod=false podCreationTimestamp="2026-07-09 14:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:16:17.594876834 +0000 UTC m=+98.199391535" watchObservedRunningTime="2026-07-09 14:16:17.595812234 +0000 UTC m=+98.200326933" Jul 09 14:16:17.608311 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.608290 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59478f8d8f-5h4rw"] Jul 09 14:16:17.624816 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.624791 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59478f8d8f-5h4rw"] Jul 09 14:16:17.702104 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.702079 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:17.803240 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.803213 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:17.911075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:17.911001 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:16:18.075208 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:18.075170 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6995c392-15cf-4a9b-8837-622f3f360654" path="/var/lib/kubelet/pods/6995c392-15cf-4a9b-8837-622f3f360654/volumes" Jul 09 14:16:20.924221 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:20.924145 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:16:22.469403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:16:22.469376 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b5ssg" Jul 09 14:17:00.890862 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:00.890825 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:00.983811 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:00.983776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:15.924824 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:15.924790 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:17:15.939794 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:15.939767 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:17:16.785779 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:16.785752 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 14:17:27.857680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:27.857646 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:28.829670 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:28.829639 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:31.474234 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:31.474149 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:31.549222 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:31.549190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:33.198246 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.198211 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:33.342925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.342892 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:33.396580 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.396541 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g82rq"] Jul 09 14:17:33.396871 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.396854 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6995c392-15cf-4a9b-8837-622f3f360654" containerName="console" Jul 09 14:17:33.396959 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.396873 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6995c392-15cf-4a9b-8837-622f3f360654" containerName="console" Jul 09 14:17:33.396959 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.396953 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6995c392-15cf-4a9b-8837-622f3f360654" containerName="console" Jul 09 14:17:33.399059 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.399037 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.401548 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.401523 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 14:17:33.401688 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.401600 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rjwf2\"" Jul 09 14:17:33.401688 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.401626 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 14:17:33.414160 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.414138 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g82rq"] Jul 09 14:17:33.483295 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.483213 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:33.496389 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.496357 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35afe813-fd0c-4f33-a409-d06232cc4c82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g82rq\" (UID: \"35afe813-fd0c-4f33-a409-d06232cc4c82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.496559 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.496451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqn2\" (UniqueName: \"kubernetes.io/projected/35afe813-fd0c-4f33-a409-d06232cc4c82-kube-api-access-9pqn2\") pod \"cert-manager-webhook-6486ffd997-g82rq\" (UID: \"35afe813-fd0c-4f33-a409-d06232cc4c82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.597595 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.597568 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35afe813-fd0c-4f33-a409-d06232cc4c82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g82rq\" (UID: \"35afe813-fd0c-4f33-a409-d06232cc4c82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.597706 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.597675 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqn2\" (UniqueName: \"kubernetes.io/projected/35afe813-fd0c-4f33-a409-d06232cc4c82-kube-api-access-9pqn2\") pod \"cert-manager-webhook-6486ffd997-g82rq\" (UID: \"35afe813-fd0c-4f33-a409-d06232cc4c82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.617518 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.617483 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35afe813-fd0c-4f33-a409-d06232cc4c82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g82rq\" (UID: \"35afe813-fd0c-4f33-a409-d06232cc4c82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.618556 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.618531 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqn2\" (UniqueName: \"kubernetes.io/projected/35afe813-fd0c-4f33-a409-d06232cc4c82-kube-api-access-9pqn2\") pod \"cert-manager-webhook-6486ffd997-g82rq\" (UID: \"35afe813-fd0c-4f33-a409-d06232cc4c82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.718382 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.718344 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:33.845967 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:33.845930 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g82rq"] Jul 09 14:17:33.848608 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:17:33.848584 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35afe813_fd0c_4f33_a409_d06232cc4c82.slice/crio-da3b92117440693d792fc391cbe07dcad12286c33e18d55f06434207aaf5db46 WatchSource:0}: Error finding container da3b92117440693d792fc391cbe07dcad12286c33e18d55f06434207aaf5db46: Status 404 returned error can't find the container with id da3b92117440693d792fc391cbe07dcad12286c33e18d55f06434207aaf5db46 Jul 09 14:17:34.831690 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:34.831656 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" event={"ID":"35afe813-fd0c-4f33-a409-d06232cc4c82","Type":"ContainerStarted","Data":"da3b92117440693d792fc391cbe07dcad12286c33e18d55f06434207aaf5db46"} Jul 09 14:17:35.338024 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:35.337989 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:35.993036 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:35.992967 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:38.252817 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:38.252778 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:38.355566 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:38.355523 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:39.853725 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:39.853687 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" event={"ID":"35afe813-fd0c-4f33-a409-d06232cc4c82","Type":"ContainerStarted","Data":"45df80f07d1627c873ddd58e3b20bd4e8da817960bbdae02b52a1f4b1a9e6e77"} Jul 09 14:17:39.853725 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:39.853731 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:39.881917 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:39.881862 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" podStartSLOduration=1.8199567509999999 podStartE2EDuration="6.881847103s" totalImagesPullingTime="5.061890352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:17:39.879753286 +0000 UTC m=+180.484267987" watchObservedRunningTime="2026-07-09 14:17:39.881847103 +0000 UTC m=+180.486361803" Jul 09 14:17:42.498589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:42.498550 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:42.602195 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:42.602159 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:43.749870 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.749838 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:43.850766 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.850740 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xdm7k"] Jul 09 14:17:43.852811 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.852794 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:43.855022 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.855005 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-svv5r\"" Jul 09 14:17:43.864337 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.864315 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xdm7k"] Jul 09 14:17:43.872919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.872900 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jzm\" (UniqueName: \"kubernetes.io/projected/12851036-cf22-4d30-ac30-2e21ada2de43-kube-api-access-d9jzm\") pod \"cert-manager-55b4d954d5-xdm7k\" (UID: \"12851036-cf22-4d30-ac30-2e21ada2de43\") " pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:43.873005 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.872966 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12851036-cf22-4d30-ac30-2e21ada2de43-bound-sa-token\") pod \"cert-manager-55b4d954d5-xdm7k\" (UID: \"12851036-cf22-4d30-ac30-2e21ada2de43\") " pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:43.973340 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.973308 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12851036-cf22-4d30-ac30-2e21ada2de43-bound-sa-token\") pod \"cert-manager-55b4d954d5-xdm7k\" (UID: \"12851036-cf22-4d30-ac30-2e21ada2de43\") " pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:43.973519 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.973352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jzm\" (UniqueName: \"kubernetes.io/projected/12851036-cf22-4d30-ac30-2e21ada2de43-kube-api-access-d9jzm\") pod \"cert-manager-55b4d954d5-xdm7k\" (UID: \"12851036-cf22-4d30-ac30-2e21ada2de43\") " pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:43.982201 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.982173 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12851036-cf22-4d30-ac30-2e21ada2de43-bound-sa-token\") pod \"cert-manager-55b4d954d5-xdm7k\" (UID: \"12851036-cf22-4d30-ac30-2e21ada2de43\") " pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:43.982333 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:43.982279 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jzm\" (UniqueName: \"kubernetes.io/projected/12851036-cf22-4d30-ac30-2e21ada2de43-kube-api-access-d9jzm\") pod \"cert-manager-55b4d954d5-xdm7k\" (UID: \"12851036-cf22-4d30-ac30-2e21ada2de43\") " pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:44.161772 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:44.161738 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xdm7k" Jul 09 14:17:44.304692 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:44.304668 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xdm7k"] Jul 09 14:17:44.307008 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:17:44.306972 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12851036_cf22_4d30_ac30_2e21ada2de43.slice/crio-d637512c81bd8eecab7f6cdfe6f60282e4df5a5740a0a1ec00d8cc2fff282693 WatchSource:0}: Error finding container d637512c81bd8eecab7f6cdfe6f60282e4df5a5740a0a1ec00d8cc2fff282693: Status 404 returned error can't find the container with id d637512c81bd8eecab7f6cdfe6f60282e4df5a5740a0a1ec00d8cc2fff282693 Jul 09 14:17:44.872517 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:44.872473 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xdm7k" event={"ID":"12851036-cf22-4d30-ac30-2e21ada2de43","Type":"ContainerStarted","Data":"2795006e216ec178352925def465a95ad396bae135b2517bb1272f9d8ac911ad"} Jul 09 14:17:44.872890 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:44.872525 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xdm7k" event={"ID":"12851036-cf22-4d30-ac30-2e21ada2de43","Type":"ContainerStarted","Data":"d637512c81bd8eecab7f6cdfe6f60282e4df5a5740a0a1ec00d8cc2fff282693"} Jul 09 14:17:44.888747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:44.888698 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xdm7k" podStartSLOduration=1.888682041 podStartE2EDuration="1.888682041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:17:44.88762701 +0000 UTC m=+185.492141711" watchObservedRunningTime="2026-07-09 14:17:44.888682041 +0000 UTC m=+185.493196741" Jul 09 14:17:45.859843 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:45.859809 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g82rq" Jul 09 14:17:49.874293 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:49.874264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:17:49.972560 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:17:49.972528 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:01.688086 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:01.688049 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:01.788954 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:01.788925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:03.387042 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:03.386996 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:05.847345 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:05.847302 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:07.661515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.661482 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:07.705586 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.705555 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn"] Jul 09 14:18:07.707777 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.707762 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.711448 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.711411 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 14:18:07.711597 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.711572 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8mjl7\"" Jul 09 14:18:07.711597 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.711592 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 14:18:07.711743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.711652 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 14:18:07.711743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.711676 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:18:07.711743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.711693 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 14:18:07.719186 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.719166 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn"] Jul 09 14:18:07.748850 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.748828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:07.751829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.751811 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0be37139-d007-4d00-b46f-60c3321be035-metrics-cert\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.751919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.751850 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg55m\" (UniqueName: \"kubernetes.io/projected/0be37139-d007-4d00-b46f-60c3321be035-kube-api-access-gg55m\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.751919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.751873 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0be37139-d007-4d00-b46f-60c3321be035-manager-config\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.752020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.751929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0be37139-d007-4d00-b46f-60c3321be035-cert\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.853282 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.853255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0be37139-d007-4d00-b46f-60c3321be035-manager-config\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.853445 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.853288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0be37139-d007-4d00-b46f-60c3321be035-cert\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.853445 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.853438 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0be37139-d007-4d00-b46f-60c3321be035-metrics-cert\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.853574 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.853476 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg55m\" (UniqueName: \"kubernetes.io/projected/0be37139-d007-4d00-b46f-60c3321be035-kube-api-access-gg55m\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.853971 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.853946 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0be37139-d007-4d00-b46f-60c3321be035-manager-config\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.855806 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.855785 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0be37139-d007-4d00-b46f-60c3321be035-cert\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.855883 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.855831 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0be37139-d007-4d00-b46f-60c3321be035-metrics-cert\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:07.865395 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:07.865365 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg55m\" (UniqueName: \"kubernetes.io/projected/0be37139-d007-4d00-b46f-60c3321be035-kube-api-access-gg55m\") pod \"lws-controller-manager-5578c64b55-7j7qn\" (UID: \"0be37139-d007-4d00-b46f-60c3321be035\") " pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:08.017580 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:08.017495 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:08.169219 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:08.169194 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn"] Jul 09 14:18:08.170936 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:18:08.170912 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be37139_d007_4d00_b46f_60c3321be035.slice/crio-497dfce415c1b6e0940628228acad1f89083efee46b5711d9267632586c2d942 WatchSource:0}: Error finding container 497dfce415c1b6e0940628228acad1f89083efee46b5711d9267632586c2d942: Status 404 returned error can't find the container with id 497dfce415c1b6e0940628228acad1f89083efee46b5711d9267632586c2d942 Jul 09 14:18:08.963966 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:08.963936 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" event={"ID":"0be37139-d007-4d00-b46f-60c3321be035","Type":"ContainerStarted","Data":"497dfce415c1b6e0940628228acad1f89083efee46b5711d9267632586c2d942"} Jul 09 14:18:15.963066 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:15.963029 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.072178 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.072142 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.166776 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.166745 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.339897 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.339870 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.447589 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.447561 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.484410 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.484379 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt"] Jul 09 14:18:16.489377 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.489351 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.491928 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.491906 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 14:18:16.492049 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.491947 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 14:18:16.492174 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.492157 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 14:18:16.492258 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.492239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 14:18:16.492861 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.492842 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qf47d\"" Jul 09 14:18:16.505203 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.505180 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt"] Jul 09 14:18:16.617525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.617453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.617525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.617496 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-manifests\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.617685 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.617530 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-tmp\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.617685 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.617569 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.617685 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.617614 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlf6\" (UniqueName: \"kubernetes.io/projected/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-kube-api-access-cnlf6\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.630029 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.630008 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.718470 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.718413 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.718607 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.718481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-manifests\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.718607 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.718533 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-tmp\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.718607 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.718563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.718607 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.718602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlf6\" (UniqueName: \"kubernetes.io/projected/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-kube-api-access-cnlf6\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.719005 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.718973 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-manifests\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.720908 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.720884 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-tmp\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.721061 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.721038 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.721119 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.721084 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.735396 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.735368 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlf6\" (UniqueName: \"kubernetes.io/projected/0836e61d-6ac8-4c08-a5af-a5d44d5213fa-kube-api-access-cnlf6\") pod \"opendatahub-operator-controller-manager-5f8568cbd5-w2fjt\" (UID: \"0836e61d-6ac8-4c08-a5af-a5d44d5213fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.736539 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.736520 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.800088 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.800065 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:16.846605 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.846537 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.923282 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.923256 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.935453 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.935414 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt"] Jul 09 14:18:16.938277 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:18:16.938252 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0836e61d_6ac8_4c08_a5af_a5d44d5213fa.slice/crio-ac3a0cb572cacffa0f75cf9d7ee2424af50bcf3b5c738d0d860325f97d2c5165 WatchSource:0}: Error finding container ac3a0cb572cacffa0f75cf9d7ee2424af50bcf3b5c738d0d860325f97d2c5165: Status 404 returned error can't find the container with id ac3a0cb572cacffa0f75cf9d7ee2424af50bcf3b5c738d0d860325f97d2c5165 Jul 09 14:18:16.991972 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.991928 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:16.992672 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:16.992653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" event={"ID":"0836e61d-6ac8-4c08-a5af-a5d44d5213fa","Type":"ContainerStarted","Data":"ac3a0cb572cacffa0f75cf9d7ee2424af50bcf3b5c738d0d860325f97d2c5165"} Jul 09 14:18:17.088499 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:17.088469 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:17.192200 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:17.192126 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:17.288179 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:17.288146 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:20.007174 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:20.007149 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" event={"ID":"0836e61d-6ac8-4c08-a5af-a5d44d5213fa","Type":"ContainerStarted","Data":"cf20e5ca6f91bc5cd382af5c4c908524068544ecd54a0ef2656497ee2b01b197"} Jul 09 14:18:22.015986 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:22.015949 2556 generic.go:356] "Generic (PLEG): container finished" podID="0836e61d-6ac8-4c08-a5af-a5d44d5213fa" containerID="cf20e5ca6f91bc5cd382af5c4c908524068544ecd54a0ef2656497ee2b01b197" exitCode=0 Jul 09 14:18:22.016356 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:22.016021 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" event={"ID":"0836e61d-6ac8-4c08-a5af-a5d44d5213fa","Type":"ContainerDied","Data":"cf20e5ca6f91bc5cd382af5c4c908524068544ecd54a0ef2656497ee2b01b197"} Jul 09 14:18:23.022232 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:23.022198 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" event={"ID":"0836e61d-6ac8-4c08-a5af-a5d44d5213fa","Type":"ContainerStarted","Data":"de601d2adec46071ecb2c34f4e762d85abb599b3773b6327352babf8bf4b7a0a"} Jul 09 14:18:23.022695 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:23.022574 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:23.048118 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:23.048068 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" podStartSLOduration=2.603869246 podStartE2EDuration="7.048051544s" totalImagesPullingTime="3.386309548s" totalInitContainerRuntime="1.05787275s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:23.046866798 +0000 UTC m=+223.651381499" watchObservedRunningTime="2026-07-09 14:18:23.048051544 +0000 UTC m=+223.652566244" Jul 09 14:18:25.030922 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:25.030887 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" event={"ID":"0be37139-d007-4d00-b46f-60c3321be035","Type":"ContainerStarted","Data":"b5792e6cf42480d9fc707f984127fb5568c210096da1e5c85abdec7b46857a67"} Jul 09 14:18:25.031328 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:25.031018 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:25.048937 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:25.048890 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" podStartSLOduration=2.125685986 podStartE2EDuration="18.048876896s" totalImagesPullingTime="15.92319091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:25.046940741 +0000 UTC m=+225.651455452" watchObservedRunningTime="2026-07-09 14:18:25.048876896 +0000 UTC m=+225.653391596" Jul 09 14:18:33.574190 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:33.574153 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:33.634914 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:33.634886 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:34.029063 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:34.028987 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8568cbd5-w2fjt" Jul 09 14:18:35.749549 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:35.749520 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:35.988768 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:35.988738 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:36.037065 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.036995 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5578c64b55-7j7qn" Jul 09 14:18:36.546195 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.546164 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:36.711098 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.711062 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:36.761327 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.761300 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-799fccbb74-28pmh"] Jul 09 14:18:36.764819 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.764792 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.767594 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.767574 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 14:18:36.767811 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.767792 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 14:18:36.768460 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.768446 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 14:18:36.768531 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.768497 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 14:18:36.768804 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.768787 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-slmz9\"" Jul 09 14:18:36.799429 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.799404 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799fccbb74-28pmh"] Jul 09 14:18:36.893775 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.893746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3c671f-941a-4f08-95bf-7102fcc80037-tls-certs\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.893968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.893801 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d3c671f-941a-4f08-95bf-7102fcc80037-tmp\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.893968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.893893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvnl\" (UniqueName: \"kubernetes.io/projected/8d3c671f-941a-4f08-95bf-7102fcc80037-kube-api-access-vfvnl\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.945707 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.945667 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:36.994557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.994523 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvnl\" (UniqueName: \"kubernetes.io/projected/8d3c671f-941a-4f08-95bf-7102fcc80037-kube-api-access-vfvnl\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.994728 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.994583 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3c671f-941a-4f08-95bf-7102fcc80037-tls-certs\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.994728 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.994620 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d3c671f-941a-4f08-95bf-7102fcc80037-tmp\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.996812 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.996789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d3c671f-941a-4f08-95bf-7102fcc80037-tmp\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:36.997045 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:36.997024 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3c671f-941a-4f08-95bf-7102fcc80037-tls-certs\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:37.009499 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:37.009474 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvnl\" (UniqueName: \"kubernetes.io/projected/8d3c671f-941a-4f08-95bf-7102fcc80037-kube-api-access-vfvnl\") pod \"kube-auth-proxy-799fccbb74-28pmh\" (UID: \"8d3c671f-941a-4f08-95bf-7102fcc80037\") " pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:37.074389 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:37.074316 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" Jul 09 14:18:37.090440 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:37.090395 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:37.177067 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:37.177039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:37.211380 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:37.211360 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799fccbb74-28pmh"] Jul 09 14:18:37.213194 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:18:37.213174 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3c671f_941a_4f08_95bf_7102fcc80037.slice/crio-a5f261076050c7138d4bc8b821ce0636acb38dec41d79fa9e304a07c2ff6078e WatchSource:0}: Error finding container a5f261076050c7138d4bc8b821ce0636acb38dec41d79fa9e304a07c2ff6078e: Status 404 returned error can't find the container with id a5f261076050c7138d4bc8b821ce0636acb38dec41d79fa9e304a07c2ff6078e Jul 09 14:18:37.275550 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:37.275525 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:37.804389 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:37.804351 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:38.071041 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:38.070958 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:38.080577 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:38.080549 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" event={"ID":"8d3c671f-941a-4f08-95bf-7102fcc80037","Type":"ContainerStarted","Data":"a5f261076050c7138d4bc8b821ce0636acb38dec41d79fa9e304a07c2ff6078e"} Jul 09 14:18:38.286989 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:38.286947 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:38.393242 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:38.393162 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:39.443779 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:39.443743 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:40.405221 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:40.405197 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 14:18:41.093872 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:41.093830 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" event={"ID":"8d3c671f-941a-4f08-95bf-7102fcc80037","Type":"ContainerStarted","Data":"abc8c54da167943737cbe8c502b7d0ef78dce13d394a9893463bf9a7d0bf714d"} Jul 09 14:18:41.113917 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:41.113867 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-799fccbb74-28pmh" podStartSLOduration=1.92568793 podStartE2EDuration="5.113851446s" totalImagesPullingTime="3.188163516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:41.1118121 +0000 UTC m=+241.716326800" watchObservedRunningTime="2026-07-09 14:18:41.113851446 +0000 UTC m=+241.718366146" Jul 09 14:18:44.110516 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:44.110485 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:44.204127 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:44.204092 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:44.976551 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:44.976517 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:45.166820 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:45.166785 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:45.256196 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:45.256124 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:47.593910 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.593877 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:47.675905 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.675867 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk"] Jul 09 14:18:47.681925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.681889 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:47.735947 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.735914 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk"] Jul 09 14:18:47.736074 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.736004 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:47.739305 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.739278 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dk59t\"" Jul 09 14:18:47.739449 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.739285 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 14:18:47.789074 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.789047 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m56jk\" (UID: \"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:47.789206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.789111 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmkp\" (UniqueName: \"kubernetes.io/projected/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-kube-api-access-5qmkp\") pod \"llmisvc-controller-manager-777c6bd7d5-m56jk\" (UID: \"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:47.890562 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.890483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m56jk\" (UID: \"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:47.890562 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.890527 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmkp\" (UniqueName: \"kubernetes.io/projected/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-kube-api-access-5qmkp\") pod \"llmisvc-controller-manager-777c6bd7d5-m56jk\" (UID: \"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:47.890790 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:18:47.890643 2556 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 09 14:18:47.890790 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:18:47.890716 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-cert podName:3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb nodeName:}" failed. No retries permitted until 2026-07-09 14:18:48.390698002 +0000 UTC m=+248.995212679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-cert") pod "llmisvc-controller-manager-777c6bd7d5-m56jk" (UID: "3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb") : secret "llmisvc-webhook-server-cert" not found Jul 09 14:18:47.909667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.909636 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmkp\" (UniqueName: \"kubernetes.io/projected/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-kube-api-access-5qmkp\") pod \"llmisvc-controller-manager-777c6bd7d5-m56jk\" (UID: \"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:47.966087 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:47.966059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:48.227937 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:48.227856 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:48.394915 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:48.394874 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m56jk\" (UID: \"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:48.397779 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:48.397747 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m56jk\" (UID: \"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:48.645990 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:48.645954 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:48.803603 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:48.803578 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk"] Jul 09 14:18:48.806179 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:18:48.806147 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d52b5de_6d7d_4dbf_bcd5_784be6cd91fb.slice/crio-1b6a99ffba63476710bfb9a4d4b8c9ab5fd31c82f26c5da828b710849c584e0a WatchSource:0}: Error finding container 1b6a99ffba63476710bfb9a4d4b8c9ab5fd31c82f26c5da828b710849c584e0a: Status 404 returned error can't find the container with id 1b6a99ffba63476710bfb9a4d4b8c9ab5fd31c82f26c5da828b710849c584e0a Jul 09 14:18:49.125125 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:49.125089 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" event={"ID":"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb","Type":"ContainerStarted","Data":"1b6a99ffba63476710bfb9a4d4b8c9ab5fd31c82f26c5da828b710849c584e0a"} Jul 09 14:18:49.759910 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:49.759870 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:49.897680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:49.897644 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:18:54.144773 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:54.144738 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" event={"ID":"3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb","Type":"ContainerStarted","Data":"aabf9a9c1ef0505bb99bd83ad212e7c62dc500c7e4c8c8caf83cdea308a09f9e"} Jul 09 14:18:54.145285 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:54.144827 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:18:54.164022 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:18:54.163974 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" podStartSLOduration=2.011499783 podStartE2EDuration="7.163959239s" totalImagesPullingTime="5.152459456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:18:54.162644724 +0000 UTC m=+254.767159448" watchObservedRunningTime="2026-07-09 14:18:54.163959239 +0000 UTC m=+254.768473980" Jul 09 14:19:12.020931 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:12.020894 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:19:25.151224 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:25.151192 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m56jk" Jul 09 14:19:37.324341 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:37.324309 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:19:37.421250 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:37.421220 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:19:38.611480 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:38.611446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:19:39.861372 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:39.861342 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:19:39.861870 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:39.861491 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:19:39.866802 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:19:39.866776 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 14:20:06.195152 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:06.195095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:08.439124 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:08.439083 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:08.492615 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:08.492576 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:11.452729 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:11.452674 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:13.596650 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:13.596598 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:14.436899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:14.436855 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:15.842557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:15.842522 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:17.433148 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:17.433108 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:18.042027 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:18.041984 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:21.076392 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:21.076361 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:21.182739 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:21.182709 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:24.186962 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:24.186929 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:24.288770 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:24.288733 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:25.807916 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.807885 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th"] Jul 09 14:20:25.810116 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.810099 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.812920 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.812891 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 09 14:20:25.813471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.813453 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gcv6t\"" Jul 09 14:20:25.813881 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.813866 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 14:20:25.814137 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.814118 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 14:20:25.817003 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.816750 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 09 14:20:25.825951 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.825932 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th"] Jul 09 14:20:25.848471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.848448 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:25.882294 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.882274 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k27q\" (UniqueName: \"kubernetes.io/projected/9dc29b52-e93a-45eb-8a26-7f532346f8e8-kube-api-access-9k27q\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.882385 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.882310 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dc29b52-e93a-45eb-8a26-7f532346f8e8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.882385 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.882375 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9dc29b52-e93a-45eb-8a26-7f532346f8e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.983470 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.983446 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k27q\" (UniqueName: \"kubernetes.io/projected/9dc29b52-e93a-45eb-8a26-7f532346f8e8-kube-api-access-9k27q\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.983584 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.983489 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dc29b52-e93a-45eb-8a26-7f532346f8e8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.983584 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.983548 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9dc29b52-e93a-45eb-8a26-7f532346f8e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.983667 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:20:25.983657 2556 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 09 14:20:25.983736 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:20:25.983725 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc29b52-e93a-45eb-8a26-7f532346f8e8-plugin-serving-cert podName:9dc29b52-e93a-45eb-8a26-7f532346f8e8 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:26.483704628 +0000 UTC m=+347.088219309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/9dc29b52-e93a-45eb-8a26-7f532346f8e8-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-jg9th" (UID: "9dc29b52-e93a-45eb-8a26-7f532346f8e8") : secret "plugin-serving-cert" not found Jul 09 14:20:25.984118 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.984099 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dc29b52-e93a-45eb-8a26-7f532346f8e8-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:25.991471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:25.991442 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k27q\" (UniqueName: \"kubernetes.io/projected/9dc29b52-e93a-45eb-8a26-7f532346f8e8-kube-api-access-9k27q\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:26.076563 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:26.076501 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:26.487355 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:26.487269 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9dc29b52-e93a-45eb-8a26-7f532346f8e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:26.489731 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:26.489711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9dc29b52-e93a-45eb-8a26-7f532346f8e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jg9th\" (UID: \"9dc29b52-e93a-45eb-8a26-7f532346f8e8\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:26.719504 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:26.719468 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" Jul 09 14:20:26.847171 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:26.847060 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th"] Jul 09 14:20:26.849802 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:20:26.849775 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dc29b52_e93a_45eb_8a26_7f532346f8e8.slice/crio-f641be90d9c457bafb88ebcf0c660d62d0fd52854aabc9f99c6c38f6a707aa65 WatchSource:0}: Error finding container f641be90d9c457bafb88ebcf0c660d62d0fd52854aabc9f99c6c38f6a707aa65: Status 404 returned error can't find the container with id f641be90d9c457bafb88ebcf0c660d62d0fd52854aabc9f99c6c38f6a707aa65 Jul 09 14:20:26.851059 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:26.851042 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:20:27.477598 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:27.477570 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:27.518435 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:27.518398 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" event={"ID":"9dc29b52-e93a-45eb-8a26-7f532346f8e8","Type":"ContainerStarted","Data":"f641be90d9c457bafb88ebcf0c660d62d0fd52854aabc9f99c6c38f6a707aa65"} Jul 09 14:20:28.809894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:28.809853 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:34.011369 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:34.011286 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:34.124035 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:34.124001 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:39.662061 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:39.662016 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:39.760958 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:39.760925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:40.138057 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:40.137997 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:41.264692 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:41.264657 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.125032 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.124991 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.193887 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.193852 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.266166 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.266135 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.354070 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.354036 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.534323 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.534245 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.635082 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.635048 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.873037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.873006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:43.972986 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:43.972953 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:47.195089 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:47.195050 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:47.294101 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:47.294058 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:51.636553 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:51.636512 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" event={"ID":"9dc29b52-e93a-45eb-8a26-7f532346f8e8","Type":"ContainerStarted","Data":"13891469c90086b025fc1498ff3dfea2956782905817bceaae138e7e32cd89fe"} Jul 09 14:20:51.654409 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:51.654358 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jg9th" podStartSLOduration=2.8969826210000003 podStartE2EDuration="26.654340662s" totalImagesPullingTime="23.757358041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:51.653981698 +0000 UTC m=+372.258496400" watchObservedRunningTime="2026-07-09 14:20:51.654340662 +0000 UTC m=+372.258855362" Jul 09 14:20:52.838898 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:52.838870 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:52.937064 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:52.937035 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:55.129617 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:55.129582 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:20:55.226459 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:20:55.226408 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:04.652013 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:04.651978 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:04.761803 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:04.761769 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:18.949724 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:18.949689 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:19.057640 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.057598 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:19.162157 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.162126 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:19.777175 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.777136 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jhjzl"] Jul 09 14:21:19.811776 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.811745 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:19.909518 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.909490 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:19.962400 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.962368 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jhjzl"] Jul 09 14:21:19.962400 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.962404 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-946b7"] Jul 09 14:21:19.962886 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.962502 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jhjzl" Jul 09 14:21:19.965318 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.965300 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wtqm6\"" Jul 09 14:21:19.973485 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.973465 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-946b7"] Jul 09 14:21:19.973583 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:19.973534 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-946b7" Jul 09 14:21:20.136684 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.136655 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6dc\" (UniqueName: \"kubernetes.io/projected/e414c6d7-612c-45ec-a7da-0de918fd1d6e-kube-api-access-lt6dc\") pod \"authorino-58cd955675-jhjzl\" (UID: \"e414c6d7-612c-45ec-a7da-0de918fd1d6e\") " pod="kuadrant-system/authorino-58cd955675-jhjzl" Jul 09 14:21:20.136829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.136689 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrzw\" (UniqueName: \"kubernetes.io/projected/4e83f270-869f-49bf-95f1-97345a92a774-kube-api-access-kmrzw\") pod \"authorino-75d556bd77-946b7\" (UID: \"4e83f270-869f-49bf-95f1-97345a92a774\") " pod="kuadrant-system/authorino-75d556bd77-946b7" Jul 09 14:21:20.237199 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.237172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6dc\" (UniqueName: \"kubernetes.io/projected/e414c6d7-612c-45ec-a7da-0de918fd1d6e-kube-api-access-lt6dc\") pod \"authorino-58cd955675-jhjzl\" (UID: \"e414c6d7-612c-45ec-a7da-0de918fd1d6e\") " pod="kuadrant-system/authorino-58cd955675-jhjzl" Jul 09 14:21:20.237368 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.237206 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrzw\" (UniqueName: \"kubernetes.io/projected/4e83f270-869f-49bf-95f1-97345a92a774-kube-api-access-kmrzw\") pod \"authorino-75d556bd77-946b7\" (UID: \"4e83f270-869f-49bf-95f1-97345a92a774\") " pod="kuadrant-system/authorino-75d556bd77-946b7" Jul 09 14:21:20.245804 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.245772 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrzw\" (UniqueName: \"kubernetes.io/projected/4e83f270-869f-49bf-95f1-97345a92a774-kube-api-access-kmrzw\") pod \"authorino-75d556bd77-946b7\" (UID: \"4e83f270-869f-49bf-95f1-97345a92a774\") " pod="kuadrant-system/authorino-75d556bd77-946b7" Jul 09 14:21:20.246087 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.246070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6dc\" (UniqueName: \"kubernetes.io/projected/e414c6d7-612c-45ec-a7da-0de918fd1d6e-kube-api-access-lt6dc\") pod \"authorino-58cd955675-jhjzl\" (UID: \"e414c6d7-612c-45ec-a7da-0de918fd1d6e\") " pod="kuadrant-system/authorino-58cd955675-jhjzl" Jul 09 14:21:20.272022 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.271994 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jhjzl" Jul 09 14:21:20.281989 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.281968 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-946b7" Jul 09 14:21:20.407527 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.407454 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jhjzl"] Jul 09 14:21:20.410992 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:21:20.410964 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode414c6d7_612c_45ec_a7da_0de918fd1d6e.slice/crio-c78879ab474fe9cca5b2cfcdd915bf7a782266a38be1d9632d073f6abd112c54 WatchSource:0}: Error finding container c78879ab474fe9cca5b2cfcdd915bf7a782266a38be1d9632d073f6abd112c54: Status 404 returned error can't find the container with id c78879ab474fe9cca5b2cfcdd915bf7a782266a38be1d9632d073f6abd112c54 Jul 09 14:21:20.431282 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.429025 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-946b7"] Jul 09 14:21:20.432586 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:21:20.432562 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e83f270_869f_49bf_95f1_97345a92a774.slice/crio-adfb84f3dc0123fdf1cd09517099b5f9447b4a7f986b26472abeca006cae0e0b WatchSource:0}: Error finding container adfb84f3dc0123fdf1cd09517099b5f9447b4a7f986b26472abeca006cae0e0b: Status 404 returned error can't find the container with id adfb84f3dc0123fdf1cd09517099b5f9447b4a7f986b26472abeca006cae0e0b Jul 09 14:21:20.756069 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.755983 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jhjzl" event={"ID":"e414c6d7-612c-45ec-a7da-0de918fd1d6e","Type":"ContainerStarted","Data":"c78879ab474fe9cca5b2cfcdd915bf7a782266a38be1d9632d073f6abd112c54"} Jul 09 14:21:20.757113 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:20.757085 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-946b7" event={"ID":"4e83f270-869f-49bf-95f1-97345a92a774","Type":"ContainerStarted","Data":"adfb84f3dc0123fdf1cd09517099b5f9447b4a7f986b26472abeca006cae0e0b"} Jul 09 14:21:24.779706 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:24.779664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jhjzl" event={"ID":"e414c6d7-612c-45ec-a7da-0de918fd1d6e","Type":"ContainerStarted","Data":"88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4"} Jul 09 14:21:24.780993 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:24.780972 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-946b7" event={"ID":"4e83f270-869f-49bf-95f1-97345a92a774","Type":"ContainerStarted","Data":"c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38"} Jul 09 14:21:24.796534 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:24.796489 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jhjzl" podStartSLOduration=1.949516783 podStartE2EDuration="5.796475792s" totalImagesPullingTime="3.846959009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:24.795025898 +0000 UTC m=+405.399540611" watchObservedRunningTime="2026-07-09 14:21:24.796475792 +0000 UTC m=+405.400990491" Jul 09 14:21:24.810134 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:24.810084 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-946b7" podStartSLOduration=1.994886857 podStartE2EDuration="5.81007133s" totalImagesPullingTime="3.815184473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:24.808236672 +0000 UTC m=+405.412751388" watchObservedRunningTime="2026-07-09 14:21:24.81007133 +0000 UTC m=+405.414586032" Jul 09 14:21:24.832084 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:24.832058 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jhjzl"] Jul 09 14:21:26.789374 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:26.789339 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jhjzl" podUID="e414c6d7-612c-45ec-a7da-0de918fd1d6e" containerName="authorino" containerID="cri-o://88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4" gracePeriod=30 Jul 09 14:21:27.038938 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.038916 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jhjzl" Jul 09 14:21:27.085555 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.085525 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt6dc\" (UniqueName: \"kubernetes.io/projected/e414c6d7-612c-45ec-a7da-0de918fd1d6e-kube-api-access-lt6dc\") pod \"e414c6d7-612c-45ec-a7da-0de918fd1d6e\" (UID: \"e414c6d7-612c-45ec-a7da-0de918fd1d6e\") " Jul 09 14:21:27.087656 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.087628 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e414c6d7-612c-45ec-a7da-0de918fd1d6e-kube-api-access-lt6dc" (OuterVolumeSpecName: "kube-api-access-lt6dc") pod "e414c6d7-612c-45ec-a7da-0de918fd1d6e" (UID: "e414c6d7-612c-45ec-a7da-0de918fd1d6e"). InnerVolumeSpecName "kube-api-access-lt6dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:21:27.186836 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.186774 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt6dc\" (UniqueName: \"kubernetes.io/projected/e414c6d7-612c-45ec-a7da-0de918fd1d6e-kube-api-access-lt6dc\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:21:27.794724 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.794690 2556 generic.go:356] "Generic (PLEG): container finished" podID="e414c6d7-612c-45ec-a7da-0de918fd1d6e" containerID="88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4" exitCode=0 Jul 09 14:21:27.795145 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.794730 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jhjzl" Jul 09 14:21:27.795145 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.794744 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jhjzl" event={"ID":"e414c6d7-612c-45ec-a7da-0de918fd1d6e","Type":"ContainerDied","Data":"88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4"} Jul 09 14:21:27.795145 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.794775 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jhjzl" event={"ID":"e414c6d7-612c-45ec-a7da-0de918fd1d6e","Type":"ContainerDied","Data":"c78879ab474fe9cca5b2cfcdd915bf7a782266a38be1d9632d073f6abd112c54"} Jul 09 14:21:27.795145 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.794791 2556 scope.go:117] "RemoveContainer" containerID="88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4" Jul 09 14:21:27.806651 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.806631 2556 scope.go:117] "RemoveContainer" containerID="88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4" Jul 09 14:21:27.806901 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:21:27.806882 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4\": container with ID starting with 88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4 not found: ID does not exist" containerID="88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4" Jul 09 14:21:27.806951 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.806910 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4"} err="failed to get container status \"88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4\": rpc error: code = NotFound desc = could not find container \"88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4\": container with ID starting with 88ae046cee6e95331d642415d02ca50c7005e9853120418f4d6f553b658119a4 not found: ID does not exist" Jul 09 14:21:27.817649 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.817630 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jhjzl"] Jul 09 14:21:27.821241 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.821220 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jhjzl"] Jul 09 14:21:27.950755 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:27.950729 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:28.075734 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:28.075705 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e414c6d7-612c-45ec-a7da-0de918fd1d6e" path="/var/lib/kubelet/pods/e414c6d7-612c-45ec-a7da-0de918fd1d6e/volumes" Jul 09 14:21:34.631680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:34.631646 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:34.797584 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:34.797560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:38.638731 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:38.638698 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:38.736984 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:38.736950 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:39.140316 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:39.140285 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:47.093581 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:47.093546 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:51.846878 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:51.846844 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:52.086549 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.086517 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sz79p"] Jul 09 14:21:52.086811 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.086798 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e414c6d7-612c-45ec-a7da-0de918fd1d6e" containerName="authorino" Jul 09 14:21:52.086857 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.086812 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e414c6d7-612c-45ec-a7da-0de918fd1d6e" containerName="authorino" Jul 09 14:21:52.086890 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.086861 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e414c6d7-612c-45ec-a7da-0de918fd1d6e" containerName="authorino" Jul 09 14:21:52.089525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.089505 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sz79p" Jul 09 14:21:52.097369 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.097312 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sz79p"] Jul 09 14:21:52.191903 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.191867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktztm\" (UniqueName: \"kubernetes.io/projected/09d8f546-74d9-4d83-a523-31feaea7c6c6-kube-api-access-ktztm\") pod \"authorino-5d4b67c446-sz79p\" (UID: \"09d8f546-74d9-4d83-a523-31feaea7c6c6\") " pod="kuadrant-system/authorino-5d4b67c446-sz79p" Jul 09 14:21:52.227939 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.227897 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:21:52.292715 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.292684 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktztm\" (UniqueName: \"kubernetes.io/projected/09d8f546-74d9-4d83-a523-31feaea7c6c6-kube-api-access-ktztm\") pod \"authorino-5d4b67c446-sz79p\" (UID: \"09d8f546-74d9-4d83-a523-31feaea7c6c6\") " pod="kuadrant-system/authorino-5d4b67c446-sz79p" Jul 09 14:21:52.302603 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.302575 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktztm\" (UniqueName: \"kubernetes.io/projected/09d8f546-74d9-4d83-a523-31feaea7c6c6-kube-api-access-ktztm\") pod \"authorino-5d4b67c446-sz79p\" (UID: \"09d8f546-74d9-4d83-a523-31feaea7c6c6\") " pod="kuadrant-system/authorino-5d4b67c446-sz79p" Jul 09 14:21:52.362883 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.362793 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sz79p"] Jul 09 14:21:52.363061 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.363049 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sz79p" Jul 09 14:21:52.390904 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.390871 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d4679d48d-6rc5c"] Jul 09 14:21:52.393253 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.393235 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4679d48d-6rc5c" Jul 09 14:21:52.401470 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.401443 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d4679d48d-6rc5c"] Jul 09 14:21:52.492699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.492674 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sz79p"] Jul 09 14:21:52.494547 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.494523 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqphc\" (UniqueName: \"kubernetes.io/projected/af185472-42df-4848-b775-72dc99e8b5e6-kube-api-access-jqphc\") pod \"authorino-d4679d48d-6rc5c\" (UID: \"af185472-42df-4848-b775-72dc99e8b5e6\") " pod="kuadrant-system/authorino-d4679d48d-6rc5c" Jul 09 14:21:52.495084 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:21:52.495056 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d8f546_74d9_4d83_a523_31feaea7c6c6.slice/crio-51af3185437c6ca4f523d7f2947d41254b09f2a43bc62c1eecb148a35a8a62e8 WatchSource:0}: Error finding container 51af3185437c6ca4f523d7f2947d41254b09f2a43bc62c1eecb148a35a8a62e8: Status 404 returned error can't find the container with id 51af3185437c6ca4f523d7f2947d41254b09f2a43bc62c1eecb148a35a8a62e8 Jul 09 14:21:52.595989 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.595950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqphc\" (UniqueName: \"kubernetes.io/projected/af185472-42df-4848-b775-72dc99e8b5e6-kube-api-access-jqphc\") pod \"authorino-d4679d48d-6rc5c\" (UID: \"af185472-42df-4848-b775-72dc99e8b5e6\") " pod="kuadrant-system/authorino-d4679d48d-6rc5c" Jul 09 14:21:52.604866 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.604843 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqphc\" (UniqueName: \"kubernetes.io/projected/af185472-42df-4848-b775-72dc99e8b5e6-kube-api-access-jqphc\") pod \"authorino-d4679d48d-6rc5c\" (UID: \"af185472-42df-4848-b775-72dc99e8b5e6\") " pod="kuadrant-system/authorino-d4679d48d-6rc5c" Jul 09 14:21:52.627966 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.627903 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4679d48d-6rc5c"] Jul 09 14:21:52.628142 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.628132 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4679d48d-6rc5c" Jul 09 14:21:52.754297 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.754272 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4679d48d-6rc5c"] Jul 09 14:21:52.756642 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:21:52.756620 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf185472_42df_4848_b775_72dc99e8b5e6.slice/crio-9d0ac9a71ed8bf6cd7b71d59eabd9096a91278291ab761a6aab37d1e44a34d34 WatchSource:0}: Error finding container 9d0ac9a71ed8bf6cd7b71d59eabd9096a91278291ab761a6aab37d1e44a34d34: Status 404 returned error can't find the container with id 9d0ac9a71ed8bf6cd7b71d59eabd9096a91278291ab761a6aab37d1e44a34d34 Jul 09 14:21:52.897651 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.897561 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sz79p" event={"ID":"09d8f546-74d9-4d83-a523-31feaea7c6c6","Type":"ContainerStarted","Data":"51af3185437c6ca4f523d7f2947d41254b09f2a43bc62c1eecb148a35a8a62e8"} Jul 09 14:21:52.898656 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:52.898631 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4679d48d-6rc5c" event={"ID":"af185472-42df-4848-b775-72dc99e8b5e6","Type":"ContainerStarted","Data":"9d0ac9a71ed8bf6cd7b71d59eabd9096a91278291ab761a6aab37d1e44a34d34"} Jul 09 14:21:53.904042 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:53.904009 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sz79p" event={"ID":"09d8f546-74d9-4d83-a523-31feaea7c6c6","Type":"ContainerStarted","Data":"71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3"} Jul 09 14:21:53.904437 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:53.904049 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-sz79p" podUID="09d8f546-74d9-4d83-a523-31feaea7c6c6" containerName="authorino" containerID="cri-o://71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3" gracePeriod=30 Jul 09 14:21:53.905583 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:53.905555 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4679d48d-6rc5c" event={"ID":"af185472-42df-4848-b775-72dc99e8b5e6","Type":"ContainerStarted","Data":"860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c"} Jul 09 14:21:53.905697 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:53.905599 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d4679d48d-6rc5c" podUID="af185472-42df-4848-b775-72dc99e8b5e6" containerName="authorino" containerID="cri-o://860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c" gracePeriod=30 Jul 09 14:21:53.922008 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:53.921968 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-sz79p" podStartSLOduration=1.375346039 podStartE2EDuration="1.921956258s" totalImagesPullingTime="546.610219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:53.920985808 +0000 UTC m=+434.525500509" watchObservedRunningTime="2026-07-09 14:21:53.921956258 +0000 UTC m=+434.526470959" Jul 09 14:21:53.936157 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:53.936115 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d4679d48d-6rc5c" podStartSLOduration=1.335725144 podStartE2EDuration="1.936101693s" totalImagesPullingTime="600.376549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:53.935066324 +0000 UTC m=+434.539581037" watchObservedRunningTime="2026-07-09 14:21:53.936101693 +0000 UTC m=+434.540616393" Jul 09 14:21:54.163564 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.163543 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4679d48d-6rc5c" Jul 09 14:21:54.166476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.166455 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sz79p" Jul 09 14:21:54.310797 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.310771 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqphc\" (UniqueName: \"kubernetes.io/projected/af185472-42df-4848-b775-72dc99e8b5e6-kube-api-access-jqphc\") pod \"af185472-42df-4848-b775-72dc99e8b5e6\" (UID: \"af185472-42df-4848-b775-72dc99e8b5e6\") " Jul 09 14:21:54.310919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.310872 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktztm\" (UniqueName: \"kubernetes.io/projected/09d8f546-74d9-4d83-a523-31feaea7c6c6-kube-api-access-ktztm\") pod \"09d8f546-74d9-4d83-a523-31feaea7c6c6\" (UID: \"09d8f546-74d9-4d83-a523-31feaea7c6c6\") " Jul 09 14:21:54.312868 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.312844 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d8f546-74d9-4d83-a523-31feaea7c6c6-kube-api-access-ktztm" (OuterVolumeSpecName: "kube-api-access-ktztm") pod "09d8f546-74d9-4d83-a523-31feaea7c6c6" (UID: "09d8f546-74d9-4d83-a523-31feaea7c6c6"). InnerVolumeSpecName "kube-api-access-ktztm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:21:54.312967 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.312875 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af185472-42df-4848-b775-72dc99e8b5e6-kube-api-access-jqphc" (OuterVolumeSpecName: "kube-api-access-jqphc") pod "af185472-42df-4848-b775-72dc99e8b5e6" (UID: "af185472-42df-4848-b775-72dc99e8b5e6"). InnerVolumeSpecName "kube-api-access-jqphc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:21:54.412441 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.412342 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqphc\" (UniqueName: \"kubernetes.io/projected/af185472-42df-4848-b775-72dc99e8b5e6-kube-api-access-jqphc\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:21:54.412441 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.412376 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktztm\" (UniqueName: \"kubernetes.io/projected/09d8f546-74d9-4d83-a523-31feaea7c6c6-kube-api-access-ktztm\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:21:54.911234 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.911202 2556 generic.go:356] "Generic (PLEG): container finished" podID="09d8f546-74d9-4d83-a523-31feaea7c6c6" containerID="71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3" exitCode=0 Jul 09 14:21:54.911676 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.911243 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sz79p" Jul 09 14:21:54.911676 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.911293 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sz79p" event={"ID":"09d8f546-74d9-4d83-a523-31feaea7c6c6","Type":"ContainerDied","Data":"71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3"} Jul 09 14:21:54.911676 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.911340 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sz79p" event={"ID":"09d8f546-74d9-4d83-a523-31feaea7c6c6","Type":"ContainerDied","Data":"51af3185437c6ca4f523d7f2947d41254b09f2a43bc62c1eecb148a35a8a62e8"} Jul 09 14:21:54.911676 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.911364 2556 scope.go:117] "RemoveContainer" containerID="71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3" Jul 09 14:21:54.912715 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.912585 2556 generic.go:356] "Generic (PLEG): container finished" podID="af185472-42df-4848-b775-72dc99e8b5e6" containerID="860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c" exitCode=0 Jul 09 14:21:54.912715 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.912618 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4679d48d-6rc5c" Jul 09 14:21:54.912715 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.912627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4679d48d-6rc5c" event={"ID":"af185472-42df-4848-b775-72dc99e8b5e6","Type":"ContainerDied","Data":"860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c"} Jul 09 14:21:54.912715 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.912654 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4679d48d-6rc5c" event={"ID":"af185472-42df-4848-b775-72dc99e8b5e6","Type":"ContainerDied","Data":"9d0ac9a71ed8bf6cd7b71d59eabd9096a91278291ab761a6aab37d1e44a34d34"} Jul 09 14:21:54.920549 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.920533 2556 scope.go:117] "RemoveContainer" containerID="71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3" Jul 09 14:21:54.920787 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:21:54.920769 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3\": container with ID starting with 71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3 not found: ID does not exist" containerID="71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3" Jul 09 14:21:54.920861 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.920799 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3"} err="failed to get container status \"71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3\": rpc error: code = NotFound desc = could not find container \"71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3\": container with ID starting with 71fa8a8f457173a72d8c64836c9720a538d963c1614d7470f9c350b2215b7dc3 not found: ID does not exist" Jul 09 14:21:54.920861 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.920823 2556 scope.go:117] "RemoveContainer" containerID="860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c" Jul 09 14:21:54.928025 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.928008 2556 scope.go:117] "RemoveContainer" containerID="860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c" Jul 09 14:21:54.928255 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:21:54.928241 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c\": container with ID starting with 860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c not found: ID does not exist" containerID="860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c" Jul 09 14:21:54.928292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.928263 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c"} err="failed to get container status \"860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c\": rpc error: code = NotFound desc = could not find container \"860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c\": container with ID starting with 860754d7e74dbce3f1b11ed58e95820df224c27774034930eb860b09ca7dd83c not found: ID does not exist" Jul 09 14:21:54.935596 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.935571 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sz79p"] Jul 09 14:21:54.939885 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.939866 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sz79p"] Jul 09 14:21:54.956154 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.956131 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4679d48d-6rc5c"] Jul 09 14:21:54.960198 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:54.960177 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d4679d48d-6rc5c"] Jul 09 14:21:56.075463 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.075413 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d8f546-74d9-4d83-a523-31feaea7c6c6" path="/var/lib/kubelet/pods/09d8f546-74d9-4d83-a523-31feaea7c6c6/volumes" Jul 09 14:21:56.075808 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.075725 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af185472-42df-4848-b775-72dc99e8b5e6" path="/var/lib/kubelet/pods/af185472-42df-4848-b775-72dc99e8b5e6/volumes" Jul 09 14:21:56.601513 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.601480 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-946b7"] Jul 09 14:21:56.601699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.601677 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-946b7" podUID="4e83f270-869f-49bf-95f1-97345a92a774" containerName="authorino" containerID="cri-o://c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38" gracePeriod=30 Jul 09 14:21:56.907477 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.907448 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-946b7" Jul 09 14:21:56.924353 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.924323 2556 generic.go:356] "Generic (PLEG): container finished" podID="4e83f270-869f-49bf-95f1-97345a92a774" containerID="c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38" exitCode=0 Jul 09 14:21:56.924493 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.924375 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-946b7" Jul 09 14:21:56.924493 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.924417 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-946b7" event={"ID":"4e83f270-869f-49bf-95f1-97345a92a774","Type":"ContainerDied","Data":"c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38"} Jul 09 14:21:56.924493 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.924472 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-946b7" event={"ID":"4e83f270-869f-49bf-95f1-97345a92a774","Type":"ContainerDied","Data":"adfb84f3dc0123fdf1cd09517099b5f9447b4a7f986b26472abeca006cae0e0b"} Jul 09 14:21:56.924629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.924493 2556 scope.go:117] "RemoveContainer" containerID="c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38" Jul 09 14:21:56.931978 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.931961 2556 scope.go:117] "RemoveContainer" containerID="c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38" Jul 09 14:21:56.932199 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:21:56.932170 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38\": container with ID starting with c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38 not found: ID does not exist" containerID="c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38" Jul 09 14:21:56.932253 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:56.932201 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38"} err="failed to get container status \"c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38\": rpc error: code = NotFound desc = could not find container \"c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38\": container with ID starting with c9a14304dfca30fafbfac87e09f2808f3c4e7859bee0fc8cc397bdd1d98f9d38 not found: ID does not exist" Jul 09 14:21:57.030260 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:57.030230 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmrzw\" (UniqueName: \"kubernetes.io/projected/4e83f270-869f-49bf-95f1-97345a92a774-kube-api-access-kmrzw\") pod \"4e83f270-869f-49bf-95f1-97345a92a774\" (UID: \"4e83f270-869f-49bf-95f1-97345a92a774\") " Jul 09 14:21:57.032194 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:57.032166 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e83f270-869f-49bf-95f1-97345a92a774-kube-api-access-kmrzw" (OuterVolumeSpecName: "kube-api-access-kmrzw") pod "4e83f270-869f-49bf-95f1-97345a92a774" (UID: "4e83f270-869f-49bf-95f1-97345a92a774"). InnerVolumeSpecName "kube-api-access-kmrzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:21:57.131914 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:57.131840 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrzw\" (UniqueName: \"kubernetes.io/projected/4e83f270-869f-49bf-95f1-97345a92a774-kube-api-access-kmrzw\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:21:57.244049 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:57.244017 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-946b7"] Jul 09 14:21:57.247537 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:57.247515 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-946b7"] Jul 09 14:21:58.074860 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:21:58.074827 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e83f270-869f-49bf-95f1-97345a92a774" path="/var/lib/kubelet/pods/4e83f270-869f-49bf-95f1-97345a92a774/volumes" Jul 09 14:22:01.108088 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.108059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:01.121769 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.121741 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f549c4c55-x5mlj"] Jul 09 14:22:01.122025 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122013 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e83f270-869f-49bf-95f1-97345a92a774" containerName="authorino" Jul 09 14:22:01.122068 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122026 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e83f270-869f-49bf-95f1-97345a92a774" containerName="authorino" Jul 09 14:22:01.122068 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122044 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d8f546-74d9-4d83-a523-31feaea7c6c6" containerName="authorino" Jul 09 14:22:01.122068 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122049 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d8f546-74d9-4d83-a523-31feaea7c6c6" containerName="authorino" Jul 09 14:22:01.122068 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122059 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af185472-42df-4848-b775-72dc99e8b5e6" containerName="authorino" Jul 09 14:22:01.122068 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122064 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af185472-42df-4848-b775-72dc99e8b5e6" containerName="authorino" Jul 09 14:22:01.122263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122107 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af185472-42df-4848-b775-72dc99e8b5e6" containerName="authorino" Jul 09 14:22:01.122263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122117 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e83f270-869f-49bf-95f1-97345a92a774" containerName="authorino" Jul 09 14:22:01.122263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.122123 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d8f546-74d9-4d83-a523-31feaea7c6c6" containerName="authorino" Jul 09 14:22:01.124816 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.124799 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.127088 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.127063 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 14:22:01.127187 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.127172 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ljgzz\"" Jul 09 14:22:01.127457 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.127440 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 14:22:01.138823 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.138666 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f549c4c55-x5mlj"] Jul 09 14:22:01.211211 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.211180 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:01.266506 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.266468 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert\") pod \"maas-controller-f549c4c55-x5mlj\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.266605 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.266547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8vt\" (UniqueName: \"kubernetes.io/projected/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-kube-api-access-dv8vt\") pod \"maas-controller-f549c4c55-x5mlj\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.276922 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.276896 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:01.336184 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.336154 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586f64b4d6-8d84g"] Jul 09 14:22:01.338546 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.338528 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.349816 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.349769 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586f64b4d6-8d84g"] Jul 09 14:22:01.367471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.367394 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert\") pod \"maas-controller-f549c4c55-x5mlj\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.367471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.367468 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8vt\" (UniqueName: \"kubernetes.io/projected/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-kube-api-access-dv8vt\") pod \"maas-controller-f549c4c55-x5mlj\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.367630 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:22:01.367514 2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 14:22:01.367630 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:22:01.367568 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert podName:3aa6ec0d-5849-408b-8ee9-e24f34cb68f2 nodeName:}" failed. No retries permitted until 2026-07-09 14:22:01.867553343 +0000 UTC m=+442.472068024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert") pod "maas-controller-f549c4c55-x5mlj" (UID: "3aa6ec0d-5849-408b-8ee9-e24f34cb68f2") : secret "maas-controller-webhook-cert" not found Jul 09 14:22:01.375662 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.375640 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8vt\" (UniqueName: \"kubernetes.io/projected/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-kube-api-access-dv8vt\") pod \"maas-controller-f549c4c55-x5mlj\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.468547 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.468508 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d96687e2-3ff2-4347-81a2-2033ba3a5610-webhook-cert\") pod \"maas-controller-586f64b4d6-8d84g\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.468547 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.468552 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjmp\" (UniqueName: \"kubernetes.io/projected/d96687e2-3ff2-4347-81a2-2033ba3a5610-kube-api-access-ttjmp\") pod \"maas-controller-586f64b4d6-8d84g\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.534324 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.534283 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:01.569703 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.569676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d96687e2-3ff2-4347-81a2-2033ba3a5610-webhook-cert\") pod \"maas-controller-586f64b4d6-8d84g\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.569816 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.569709 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjmp\" (UniqueName: \"kubernetes.io/projected/d96687e2-3ff2-4347-81a2-2033ba3a5610-kube-api-access-ttjmp\") pod \"maas-controller-586f64b4d6-8d84g\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.572004 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.571977 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d96687e2-3ff2-4347-81a2-2033ba3a5610-webhook-cert\") pod \"maas-controller-586f64b4d6-8d84g\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.577268 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.577243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjmp\" (UniqueName: \"kubernetes.io/projected/d96687e2-3ff2-4347-81a2-2033ba3a5610-kube-api-access-ttjmp\") pod \"maas-controller-586f64b4d6-8d84g\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.648393 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.648312 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:01.777131 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.777110 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586f64b4d6-8d84g"] Jul 09 14:22:01.779758 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:22:01.779732 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96687e2_3ff2_4347_81a2_2033ba3a5610.slice/crio-8dce125c3431f59c6a1e983adf890533f3a6a5613f0bde02f9ea8689b6f37f5c WatchSource:0}: Error finding container 8dce125c3431f59c6a1e983adf890533f3a6a5613f0bde02f9ea8689b6f37f5c: Status 404 returned error can't find the container with id 8dce125c3431f59c6a1e983adf890533f3a6a5613f0bde02f9ea8689b6f37f5c Jul 09 14:22:01.872375 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.872347 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert\") pod \"maas-controller-f549c4c55-x5mlj\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.874617 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.874599 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert\") pod \"maas-controller-f549c4c55-x5mlj\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:01.947742 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:01.947670 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586f64b4d6-8d84g" event={"ID":"d96687e2-3ff2-4347-81a2-2033ba3a5610","Type":"ContainerStarted","Data":"8dce125c3431f59c6a1e983adf890533f3a6a5613f0bde02f9ea8689b6f37f5c"} Jul 09 14:22:02.035208 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:02.035176 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:02.163168 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:02.163142 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f549c4c55-x5mlj"] Jul 09 14:22:02.165657 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:22:02.165631 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa6ec0d_5849_408b_8ee9_e24f34cb68f2.slice/crio-9430ac91c7aef92830cd9351843911d04001a916cccc2f7ad9ea692941a4e73d WatchSource:0}: Error finding container 9430ac91c7aef92830cd9351843911d04001a916cccc2f7ad9ea692941a4e73d: Status 404 returned error can't find the container with id 9430ac91c7aef92830cd9351843911d04001a916cccc2f7ad9ea692941a4e73d Jul 09 14:22:02.956478 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:02.956386 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f549c4c55-x5mlj" event={"ID":"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2","Type":"ContainerStarted","Data":"9430ac91c7aef92830cd9351843911d04001a916cccc2f7ad9ea692941a4e73d"} Jul 09 14:22:04.422434 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:04.422402 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:04.968091 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:04.968059 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f549c4c55-x5mlj" event={"ID":"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2","Type":"ContainerStarted","Data":"147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6"} Jul 09 14:22:04.968264 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:04.968151 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:04.969466 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:04.969437 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586f64b4d6-8d84g" event={"ID":"d96687e2-3ff2-4347-81a2-2033ba3a5610","Type":"ContainerStarted","Data":"821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed"} Jul 09 14:22:04.969559 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:04.969544 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:04.992076 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:04.992029 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f549c4c55-x5mlj" podStartSLOduration=1.949395919 podStartE2EDuration="3.992016431s" totalImagesPullingTime="2.042620512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:04.990490743 +0000 UTC m=+445.595005457" watchObservedRunningTime="2026-07-09 14:22:04.992016431 +0000 UTC m=+445.596531131" Jul 09 14:22:05.008436 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:05.008382 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-586f64b4d6-8d84g" podStartSLOduration=1.589026079 podStartE2EDuration="4.008366416s" totalImagesPullingTime="2.419340337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:05.008088495 +0000 UTC m=+445.612603194" watchObservedRunningTime="2026-07-09 14:22:05.008366416 +0000 UTC m=+445.612881116" Jul 09 14:22:05.166457 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:05.166408 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:15.979534 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:15.979502 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:15.979924 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:15.979566 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:22:16.027372 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.027337 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f549c4c55-x5mlj"] Jul 09 14:22:16.027598 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.027571 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f549c4c55-x5mlj" podUID="3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" containerName="manager" containerID="cri-o://147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6" gracePeriod=30 Jul 09 14:22:16.268315 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.268285 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:16.295676 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.295645 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv8vt\" (UniqueName: \"kubernetes.io/projected/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-kube-api-access-dv8vt\") pod \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " Jul 09 14:22:16.295826 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.295697 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert\") pod \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\" (UID: \"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2\") " Jul 09 14:22:16.297874 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.297843 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" (UID: "3aa6ec0d-5849-408b-8ee9-e24f34cb68f2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:22:16.297986 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.297958 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-kube-api-access-dv8vt" (OuterVolumeSpecName: "kube-api-access-dv8vt") pod "3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" (UID: "3aa6ec0d-5849-408b-8ee9-e24f34cb68f2"). InnerVolumeSpecName "kube-api-access-dv8vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:22:16.396688 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.396657 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv8vt\" (UniqueName: \"kubernetes.io/projected/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-kube-api-access-dv8vt\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:22:16.396688 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:16.396682 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2-webhook-cert\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:22:17.030374 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.030337 2556 generic.go:356] "Generic (PLEG): container finished" podID="3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" containerID="147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6" exitCode=0 Jul 09 14:22:17.030774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.030385 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f549c4c55-x5mlj" Jul 09 14:22:17.030774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.030417 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f549c4c55-x5mlj" event={"ID":"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2","Type":"ContainerDied","Data":"147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6"} Jul 09 14:22:17.030774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.030490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f549c4c55-x5mlj" event={"ID":"3aa6ec0d-5849-408b-8ee9-e24f34cb68f2","Type":"ContainerDied","Data":"9430ac91c7aef92830cd9351843911d04001a916cccc2f7ad9ea692941a4e73d"} Jul 09 14:22:17.030774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.030512 2556 scope.go:117] "RemoveContainer" containerID="147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6" Jul 09 14:22:17.039068 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.039052 2556 scope.go:117] "RemoveContainer" containerID="147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6" Jul 09 14:22:17.039310 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:22:17.039295 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6\": container with ID starting with 147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6 not found: ID does not exist" containerID="147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6" Jul 09 14:22:17.039351 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.039317 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6"} err="failed to get container status \"147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6\": rpc error: code = NotFound desc = could not find container \"147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6\": container with ID starting with 147b0df3f3b3757025c5ebae241fd50db58a45ebcc862c50f9aa9411574498f6 not found: ID does not exist" Jul 09 14:22:17.052478 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.052455 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f549c4c55-x5mlj"] Jul 09 14:22:17.059572 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:17.059552 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f549c4c55-x5mlj"] Jul 09 14:22:18.078061 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:18.078027 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" path="/var/lib/kubelet/pods/3aa6ec0d-5849-408b-8ee9-e24f34cb68f2/volumes" Jul 09 14:22:34.910368 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:34.910321 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:35.316290 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.316264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:35.377779 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.377742 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5"] Jul 09 14:22:35.378056 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.378043 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" containerName="manager" Jul 09 14:22:35.378056 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.378056 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" containerName="manager" Jul 09 14:22:35.378191 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.378105 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa6ec0d-5849-408b-8ee9-e24f34cb68f2" containerName="manager" Jul 09 14:22:35.380733 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.380711 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.383158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.383132 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 14:22:35.383158 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.383154 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 14:22:35.383335 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.383187 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 14:22:35.383899 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.383880 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bmdnz\"" Jul 09 14:22:35.391600 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.391576 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5"] Jul 09 14:22:35.430767 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.430740 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-maas-api-tls\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.430870 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.430796 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8gf\" (UniqueName: \"kubernetes.io/projected/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-kube-api-access-cw8gf\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.430870 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.430844 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-openshift-service-ca\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.507219 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.507192 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:35.531489 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.531469 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-maas-api-tls\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.531573 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.531515 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8gf\" (UniqueName: \"kubernetes.io/projected/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-kube-api-access-cw8gf\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.531573 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.531542 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-openshift-service-ca\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.532136 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.532120 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-openshift-service-ca\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.533757 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.533733 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-maas-api-tls\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.540355 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.540336 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8gf\" (UniqueName: \"kubernetes.io/projected/a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7-kube-api-access-cw8gf\") pod \"maas-api-7cfbd6c584-qsmm5\" (UID: \"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7\") " pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.692345 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.692277 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:35.827890 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.827854 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5"] Jul 09 14:22:35.831148 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:22:35.831119 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4dafa70_6e4b_4eaf_b492_3653b2e3b2e7.slice/crio-162fde6397ee0c51f8c767c26214f1791323e6d6e06284af09e52016f1e4eed1 WatchSource:0}: Error finding container 162fde6397ee0c51f8c767c26214f1791323e6d6e06284af09e52016f1e4eed1: Status 404 returned error can't find the container with id 162fde6397ee0c51f8c767c26214f1791323e6d6e06284af09e52016f1e4eed1 Jul 09 14:22:35.846230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.846201 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:35.946857 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.946787 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:35.953319 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.953296 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-vjmxz"] Jul 09 14:22:35.955995 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.955976 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:35.960497 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.960478 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vgcj9\"" Jul 09 14:22:35.961044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.961028 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 14:22:35.972379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.972354 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-vjmxz"] Jul 09 14:22:35.993298 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.993273 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-ms9cp"] Jul 09 14:22:35.995529 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:35.995515 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.013390 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.013367 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-ms9cp"] Jul 09 14:22:36.035674 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.035643 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfbj\" (UniqueName: \"kubernetes.io/projected/b072044b-0a70-4c8b-a3de-a8ecab3cf841-kube-api-access-scfbj\") pod \"payload-processing-58f86c7798-ms9cp\" (UID: \"b072044b-0a70-4c8b-a3de-a8ecab3cf841\") " pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.035771 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.035680 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b67823-9b07-449f-a97a-547a3c2ac4c3-plugins-config-volume\") pod \"payload-pre-processing-7964457696-vjmxz\" (UID: \"f8b67823-9b07-449f-a97a-547a3c2ac4c3\") " pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:36.035771 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.035719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5p8q\" (UniqueName: \"kubernetes.io/projected/f8b67823-9b07-449f-a97a-547a3c2ac4c3-kube-api-access-w5p8q\") pod \"payload-pre-processing-7964457696-vjmxz\" (UID: \"f8b67823-9b07-449f-a97a-547a3c2ac4c3\") " pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:36.035771 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.035739 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072044b-0a70-4c8b-a3de-a8ecab3cf841-plugins-config-volume\") pod \"payload-processing-58f86c7798-ms9cp\" (UID: \"b072044b-0a70-4c8b-a3de-a8ecab3cf841\") " pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.111829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.111796 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" event={"ID":"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7","Type":"ContainerStarted","Data":"162fde6397ee0c51f8c767c26214f1791323e6d6e06284af09e52016f1e4eed1"} Jul 09 14:22:36.136434 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.136386 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfbj\" (UniqueName: \"kubernetes.io/projected/b072044b-0a70-4c8b-a3de-a8ecab3cf841-kube-api-access-scfbj\") pod \"payload-processing-58f86c7798-ms9cp\" (UID: \"b072044b-0a70-4c8b-a3de-a8ecab3cf841\") " pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.136585 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.136440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b67823-9b07-449f-a97a-547a3c2ac4c3-plugins-config-volume\") pod \"payload-pre-processing-7964457696-vjmxz\" (UID: \"f8b67823-9b07-449f-a97a-547a3c2ac4c3\") " pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:36.136585 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.136477 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5p8q\" (UniqueName: \"kubernetes.io/projected/f8b67823-9b07-449f-a97a-547a3c2ac4c3-kube-api-access-w5p8q\") pod \"payload-pre-processing-7964457696-vjmxz\" (UID: \"f8b67823-9b07-449f-a97a-547a3c2ac4c3\") " pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:36.136585 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.136508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072044b-0a70-4c8b-a3de-a8ecab3cf841-plugins-config-volume\") pod \"payload-processing-58f86c7798-ms9cp\" (UID: \"b072044b-0a70-4c8b-a3de-a8ecab3cf841\") " pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.137283 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.137260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b67823-9b07-449f-a97a-547a3c2ac4c3-plugins-config-volume\") pod \"payload-pre-processing-7964457696-vjmxz\" (UID: \"f8b67823-9b07-449f-a97a-547a3c2ac4c3\") " pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:36.137358 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.137344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072044b-0a70-4c8b-a3de-a8ecab3cf841-plugins-config-volume\") pod \"payload-processing-58f86c7798-ms9cp\" (UID: \"b072044b-0a70-4c8b-a3de-a8ecab3cf841\") " pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.144897 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.144871 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5p8q\" (UniqueName: \"kubernetes.io/projected/f8b67823-9b07-449f-a97a-547a3c2ac4c3-kube-api-access-w5p8q\") pod \"payload-pre-processing-7964457696-vjmxz\" (UID: \"f8b67823-9b07-449f-a97a-547a3c2ac4c3\") " pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:36.144980 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.144899 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfbj\" (UniqueName: \"kubernetes.io/projected/b072044b-0a70-4c8b-a3de-a8ecab3cf841-kube-api-access-scfbj\") pod \"payload-processing-58f86c7798-ms9cp\" (UID: \"b072044b-0a70-4c8b-a3de-a8ecab3cf841\") " pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.271269 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.271196 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:36.305324 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.305294 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:36.407704 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.407674 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-vjmxz"] Jul 09 14:22:36.410064 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:22:36.410037 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b67823_9b07_449f_a97a_547a3c2ac4c3.slice/crio-f956543602f45a3122f6f125a17e730de1a161f01e49cedd158a19f413f6e745 WatchSource:0}: Error finding container f956543602f45a3122f6f125a17e730de1a161f01e49cedd158a19f413f6e745: Status 404 returned error can't find the container with id f956543602f45a3122f6f125a17e730de1a161f01e49cedd158a19f413f6e745 Jul 09 14:22:36.444756 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:36.443777 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-ms9cp"] Jul 09 14:22:36.446762 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:22:36.446730 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb072044b_0a70_4c8b_a3de_a8ecab3cf841.slice/crio-7cee27b5dc2f5dc40985cc5fc2d0dcf3c6cf279e6ac2aebf264e7ea2ada4cc56 WatchSource:0}: Error finding container 7cee27b5dc2f5dc40985cc5fc2d0dcf3c6cf279e6ac2aebf264e7ea2ada4cc56: Status 404 returned error can't find the container with id 7cee27b5dc2f5dc40985cc5fc2d0dcf3c6cf279e6ac2aebf264e7ea2ada4cc56 Jul 09 14:22:37.117481 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:37.117442 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" event={"ID":"f8b67823-9b07-449f-a97a-547a3c2ac4c3","Type":"ContainerStarted","Data":"f956543602f45a3122f6f125a17e730de1a161f01e49cedd158a19f413f6e745"} Jul 09 14:22:37.118750 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:37.118726 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" event={"ID":"b072044b-0a70-4c8b-a3de-a8ecab3cf841","Type":"ContainerStarted","Data":"7cee27b5dc2f5dc40985cc5fc2d0dcf3c6cf279e6ac2aebf264e7ea2ada4cc56"} Jul 09 14:22:38.127111 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:38.127067 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" event={"ID":"a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7","Type":"ContainerStarted","Data":"04e9a1306509be582d8d957910abeabcb140e20820f20b098ed5fb1887886bb9"} Jul 09 14:22:38.127591 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:38.127285 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:38.145835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:38.145786 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" podStartSLOduration=1.424682993 podStartE2EDuration="3.145765999s" totalImagesPullingTime="1.721083006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:38.143467141 +0000 UTC m=+478.747981843" watchObservedRunningTime="2026-07-09 14:22:38.145765999 +0000 UTC m=+478.750280702" Jul 09 14:22:39.133330 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:39.133294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" event={"ID":"f8b67823-9b07-449f-a97a-547a3c2ac4c3","Type":"ContainerStarted","Data":"bf216866294814795046d92652781ff445ac9926d6c2e200da5abeb8e961d5e2"} Jul 09 14:22:39.133761 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:39.133440 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:39.134877 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:39.134840 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" event={"ID":"b072044b-0a70-4c8b-a3de-a8ecab3cf841","Type":"ContainerStarted","Data":"6ece6c866b8c179f72f57be2524b0704e629d13439d10df3b9ad1e1158605dc5"} Jul 09 14:22:39.151311 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:39.151263 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" podStartSLOduration=1.9977349389999999 podStartE2EDuration="4.151247359s" totalImagesPullingTime="2.15351242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:39.149628858 +0000 UTC m=+479.754143559" watchObservedRunningTime="2026-07-09 14:22:39.151247359 +0000 UTC m=+479.755762058" Jul 09 14:22:39.165410 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:39.165370 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" podStartSLOduration=2.050424079 podStartE2EDuration="4.165356947s" totalImagesPullingTime="2.114932868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:39.163622313 +0000 UTC m=+479.768137014" watchObservedRunningTime="2026-07-09 14:22:39.165356947 +0000 UTC m=+479.769871646" Jul 09 14:22:40.140114 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:40.140082 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:22:44.140915 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:44.140883 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cfbd6c584-qsmm5" Jul 09 14:22:45.163809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:45.163781 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:45.259693 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:45.259663 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:22:50.142400 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:50.142368 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-vjmxz" Jul 09 14:22:51.145809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:22:51.145779 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-ms9cp" Jul 09 14:23:01.230596 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:01.230565 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:12.774387 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.774349 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q"] Jul 09 14:23:12.777472 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.777450 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:12.780356 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.780337 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 14:23:12.781042 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.781017 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 14:23:12.781122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.781069 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 09 14:23:12.781122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.781095 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rmz94\"" Jul 09 14:23:12.791174 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.791154 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q"] Jul 09 14:23:12.811878 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.811850 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:12.954904 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.954867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:12.954904 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.954911 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:12.955100 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.954943 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4cv\" (UniqueName: \"kubernetes.io/projected/32a556da-20a5-4f2c-b606-125329c25035-kube-api-access-kk4cv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:12.955100 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.955012 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:12.955100 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.955029 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a556da-20a5-4f2c-b606-125329c25035-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:12.955100 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:12.955043 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056227 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056364 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056235 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a556da-20a5-4f2c-b606-125329c25035-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056364 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056253 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056468 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056402 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056468 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056462 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056517 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4cv\" (UniqueName: \"kubernetes.io/projected/32a556da-20a5-4f2c-b606-125329c25035-kube-api-access-kk4cv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056658 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.056793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.056734 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.058639 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.058623 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a556da-20a5-4f2c-b606-125329c25035-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.058763 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.058749 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a556da-20a5-4f2c-b606-125329c25035-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.065277 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.065252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4cv\" (UniqueName: \"kubernetes.io/projected/32a556da-20a5-4f2c-b606-125329c25035-kube-api-access-kk4cv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vbz7q\" (UID: \"32a556da-20a5-4f2c-b606-125329c25035\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.087809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.087784 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:13.245344 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.245321 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q"] Jul 09 14:23:13.247579 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:23:13.247549 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a556da_20a5_4f2c_b606_125329c25035.slice/crio-e8ff176af43830173fa3f456ad70dc36c27b140cf76950838d27bb87622f121a WatchSource:0}: Error finding container e8ff176af43830173fa3f456ad70dc36c27b140cf76950838d27bb87622f121a: Status 404 returned error can't find the container with id e8ff176af43830173fa3f456ad70dc36c27b140cf76950838d27bb87622f121a Jul 09 14:23:13.289137 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:13.289106 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" event={"ID":"32a556da-20a5-4f2c-b606-125329c25035","Type":"ContainerStarted","Data":"e8ff176af43830173fa3f456ad70dc36c27b140cf76950838d27bb87622f121a"} Jul 09 14:23:15.299785 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:15.299750 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" event={"ID":"32a556da-20a5-4f2c-b606-125329c25035","Type":"ContainerStarted","Data":"47ccb9fd33a10ce0efec4200e81a93f6e697df52f388664b20becc04bb0a4ea6"} Jul 09 14:23:15.332842 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:15.332794 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" podStartSLOduration=2.160113329 podStartE2EDuration="3.332776801s" totalImagesPullingTime="1.172663472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:23:15.331387304 +0000 UTC m=+515.935902003" watchObservedRunningTime="2026-07-09 14:23:15.332776801 +0000 UTC m=+515.937291502" Jul 09 14:23:18.214417 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:18.214374 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:23.088829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.088792 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:23.089216 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.088853 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:23.101641 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.101612 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:23.347344 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.347256 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vbz7q" Jul 09 14:23:23.490672 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.490640 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z"] Jul 09 14:23:23.492994 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.492975 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.495297 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.495274 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 14:23:23.506353 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.506330 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z"] Jul 09 14:23:23.535361 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.535333 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:23.553533 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.553505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.553533 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.553535 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.553697 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.553561 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.553697 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.553588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc4b5\" (UniqueName: \"kubernetes.io/projected/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-kube-api-access-gc4b5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.553697 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.553611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.553697 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.553658 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.654544 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.654474 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.654544 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.654509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.654544 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.654529 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.654793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.654553 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4b5\" (UniqueName: \"kubernetes.io/projected/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-kube-api-access-gc4b5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.654793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.654577 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.654793 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.654604 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.655006 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.654970 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.655137 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.655016 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.655137 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.655092 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.656803 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.656779 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.657062 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.657045 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.663568 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.663539 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4b5\" (UniqueName: \"kubernetes.io/projected/d2720ab2-bba6-4bf5-84d4-55940f8f08f3-kube-api-access-gc4b5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z\" (UID: \"d2720ab2-bba6-4bf5-84d4-55940f8f08f3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.803558 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.803521 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:23.933854 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:23.933767 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z"] Jul 09 14:23:23.936104 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:23:23.936073 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2720ab2_bba6_4bf5_84d4_55940f8f08f3.slice/crio-a17047e0958b67222be9d2464e6922290a7f09cf3e93079fa9ec60e9980556c7 WatchSource:0}: Error finding container a17047e0958b67222be9d2464e6922290a7f09cf3e93079fa9ec60e9980556c7: Status 404 returned error can't find the container with id a17047e0958b67222be9d2464e6922290a7f09cf3e93079fa9ec60e9980556c7 Jul 09 14:23:24.341391 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:24.341346 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" event={"ID":"d2720ab2-bba6-4bf5-84d4-55940f8f08f3","Type":"ContainerStarted","Data":"b47de33d52c011de09440bf161afb951c8602568e46dd909a5e7d17cd5e9eceb"} Jul 09 14:23:24.341797 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:24.341415 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" event={"ID":"d2720ab2-bba6-4bf5-84d4-55940f8f08f3","Type":"ContainerStarted","Data":"a17047e0958b67222be9d2464e6922290a7f09cf3e93079fa9ec60e9980556c7"} Jul 09 14:23:24.371488 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:24.371438 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" podStartSLOduration=1.1639478859999999 podStartE2EDuration="1.371410267s" totalImagesPullingTime="207.462381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:23:24.369878151 +0000 UTC m=+524.974392851" watchObservedRunningTime="2026-07-09 14:23:24.371410267 +0000 UTC m=+524.975924966" Jul 09 14:23:32.128898 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.128807 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:32.200479 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.200445 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:32.386676 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.386599 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49"] Jul 09 14:23:32.389604 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.389580 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.392918 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.392901 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 09 14:23:32.402058 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.402037 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49"] Jul 09 14:23:32.404875 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.404841 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:32.433964 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.433937 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.434078 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.434052 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.434151 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.434083 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7c5\" (UniqueName: \"kubernetes.io/projected/cc49f76b-c7b1-49be-b621-05f71623859e-kube-api-access-5h7c5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.434214 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.434156 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.434214 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.434205 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc49f76b-c7b1-49be-b621-05f71623859e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.434307 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.434248 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.534894 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.534863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.534914 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.534931 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7c5\" (UniqueName: \"kubernetes.io/projected/cc49f76b-c7b1-49be-b621-05f71623859e-kube-api-access-5h7c5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.534950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.534969 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc49f76b-c7b1-49be-b621-05f71623859e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.534992 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535311 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.535291 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535378 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.535358 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.535446 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.535408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.537262 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.537238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc49f76b-c7b1-49be-b621-05f71623859e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.537357 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.537331 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc49f76b-c7b1-49be-b621-05f71623859e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.547047 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.547025 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7c5\" (UniqueName: \"kubernetes.io/projected/cc49f76b-c7b1-49be-b621-05f71623859e-kube-api-access-5h7c5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lxc49\" (UID: \"cc49f76b-c7b1-49be-b621-05f71623859e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.700084 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.700019 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:32.829871 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:32.829848 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49"] Jul 09 14:23:32.832288 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:23:32.832263 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc49f76b_c7b1_49be_b621_05f71623859e.slice/crio-1da6bc231b8353a37c013afab38639aece85906fb9840afe0035098daeb448ca WatchSource:0}: Error finding container 1da6bc231b8353a37c013afab38639aece85906fb9840afe0035098daeb448ca: Status 404 returned error can't find the container with id 1da6bc231b8353a37c013afab38639aece85906fb9840afe0035098daeb448ca Jul 09 14:23:33.382633 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:33.382546 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" event={"ID":"cc49f76b-c7b1-49be-b621-05f71623859e","Type":"ContainerStarted","Data":"6edc0b3acf8f05008a7a81b2d09c0a4fc52973a0959debb32d304ec25d8ee9c1"} Jul 09 14:23:33.382633 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:33.382600 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" event={"ID":"cc49f76b-c7b1-49be-b621-05f71623859e","Type":"ContainerStarted","Data":"1da6bc231b8353a37c013afab38639aece85906fb9840afe0035098daeb448ca"} Jul 09 14:23:33.405536 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:33.405478 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" podStartSLOduration=1.163381505 podStartE2EDuration="1.405465206s" totalImagesPullingTime="242.083701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:23:33.403660302 +0000 UTC m=+534.008175000" watchObservedRunningTime="2026-07-09 14:23:33.405465206 +0000 UTC m=+534.009979928" Jul 09 14:23:33.803829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:33.803803 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:33.803975 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:33.803875 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:33.815869 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:33.815846 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:34.398995 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:34.398962 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z" Jul 09 14:23:36.909861 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:36.909822 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:40.611943 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:40.611909 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:42.700338 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:42.700297 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:42.700725 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:42.700387 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:42.712686 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:42.712661 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:23:43.081018 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:43.080983 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:23:43.436980 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:23:43.436903 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lxc49" Jul 09 14:24:03.765206 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:03.765169 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:24:19.302869 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.302837 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b6bbc4f8f-km5k9"] Jul 09 14:24:19.306196 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.306177 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.309008 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.308985 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 14:24:19.310915 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.310896 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wtqm6\"" Jul 09 14:24:19.318251 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.318231 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b6bbc4f8f-km5k9"] Jul 09 14:24:19.434356 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.434333 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e078cc7-84e6-4f96-9520-663490fad786-tls-cert\") pod \"authorino-5b6bbc4f8f-km5k9\" (UID: \"7e078cc7-84e6-4f96-9520-663490fad786\") " pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.434515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.434361 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2th\" (UniqueName: \"kubernetes.io/projected/7e078cc7-84e6-4f96-9520-663490fad786-kube-api-access-8x2th\") pod \"authorino-5b6bbc4f8f-km5k9\" (UID: \"7e078cc7-84e6-4f96-9520-663490fad786\") " pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.535761 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.535726 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e078cc7-84e6-4f96-9520-663490fad786-tls-cert\") pod \"authorino-5b6bbc4f8f-km5k9\" (UID: \"7e078cc7-84e6-4f96-9520-663490fad786\") " pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.535761 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.535763 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2th\" (UniqueName: \"kubernetes.io/projected/7e078cc7-84e6-4f96-9520-663490fad786-kube-api-access-8x2th\") pod \"authorino-5b6bbc4f8f-km5k9\" (UID: \"7e078cc7-84e6-4f96-9520-663490fad786\") " pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.538196 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.538165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e078cc7-84e6-4f96-9520-663490fad786-tls-cert\") pod \"authorino-5b6bbc4f8f-km5k9\" (UID: \"7e078cc7-84e6-4f96-9520-663490fad786\") " pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.546101 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.546075 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2th\" (UniqueName: \"kubernetes.io/projected/7e078cc7-84e6-4f96-9520-663490fad786-kube-api-access-8x2th\") pod \"authorino-5b6bbc4f8f-km5k9\" (UID: \"7e078cc7-84e6-4f96-9520-663490fad786\") " pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.615122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.615037 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" Jul 09 14:24:19.687262 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.687173 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:24:19.752171 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:19.752138 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b6bbc4f8f-km5k9"] Jul 09 14:24:19.753872 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:24:19.753844 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e078cc7_84e6_4f96_9520_663490fad786.slice/crio-cd1e3314566d587761273501dcba9824a1ee53e6e2b39d9e8a7a40786f6b50bb WatchSource:0}: Error finding container cd1e3314566d587761273501dcba9824a1ee53e6e2b39d9e8a7a40786f6b50bb: Status 404 returned error can't find the container with id cd1e3314566d587761273501dcba9824a1ee53e6e2b39d9e8a7a40786f6b50bb Jul 09 14:24:20.595542 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:20.595503 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" event={"ID":"7e078cc7-84e6-4f96-9520-663490fad786","Type":"ContainerStarted","Data":"09fab7c97d53d5c7647f0aafa1001b8033f5da93c7dbdb13f57936bafd3b84ca"} Jul 09 14:24:20.595928 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:20.595552 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" event={"ID":"7e078cc7-84e6-4f96-9520-663490fad786","Type":"ContainerStarted","Data":"cd1e3314566d587761273501dcba9824a1ee53e6e2b39d9e8a7a40786f6b50bb"} Jul 09 14:24:20.641207 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:20.641144 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b6bbc4f8f-km5k9" podStartSLOduration=1.243650624 podStartE2EDuration="1.641110394s" totalImagesPullingTime="397.45977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:24:20.632086697 +0000 UTC m=+581.236601398" watchObservedRunningTime="2026-07-09 14:24:20.641110394 +0000 UTC m=+581.245625097" Jul 09 14:24:21.258574 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:21.258532 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:24:39.887569 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:39.887541 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:24:39.888586 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:39.888566 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:24:40.790619 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:40.790582 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:24:45.063295 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:24:45.063235 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:25:27.979694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:27.979656 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:25:28.078020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:28.077969 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:25:30.835344 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:30.835301 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:25:30.902129 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:30.902061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:25:51.949759 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:51.949725 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:25:52.058936 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.058896 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586f64b4d6-8d84g"] Jul 09 14:25:52.059190 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.059150 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-586f64b4d6-8d84g" podUID="d96687e2-3ff2-4347-81a2-2033ba3a5610" containerName="manager" containerID="cri-o://821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed" gracePeriod=30 Jul 09 14:25:52.310500 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.310475 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:25:52.340922 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.340890 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttjmp\" (UniqueName: \"kubernetes.io/projected/d96687e2-3ff2-4347-81a2-2033ba3a5610-kube-api-access-ttjmp\") pod \"d96687e2-3ff2-4347-81a2-2033ba3a5610\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " Jul 09 14:25:52.341084 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.340975 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d96687e2-3ff2-4347-81a2-2033ba3a5610-webhook-cert\") pod \"d96687e2-3ff2-4347-81a2-2033ba3a5610\" (UID: \"d96687e2-3ff2-4347-81a2-2033ba3a5610\") " Jul 09 14:25:52.343153 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.343118 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96687e2-3ff2-4347-81a2-2033ba3a5610-kube-api-access-ttjmp" (OuterVolumeSpecName: "kube-api-access-ttjmp") pod "d96687e2-3ff2-4347-81a2-2033ba3a5610" (UID: "d96687e2-3ff2-4347-81a2-2033ba3a5610"). InnerVolumeSpecName "kube-api-access-ttjmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:25:52.343263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.343224 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96687e2-3ff2-4347-81a2-2033ba3a5610-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d96687e2-3ff2-4347-81a2-2033ba3a5610" (UID: "d96687e2-3ff2-4347-81a2-2033ba3a5610"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:25:52.442011 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.441983 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d96687e2-3ff2-4347-81a2-2033ba3a5610-webhook-cert\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:25:52.442011 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:52.442008 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttjmp\" (UniqueName: \"kubernetes.io/projected/d96687e2-3ff2-4347-81a2-2033ba3a5610-kube-api-access-ttjmp\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:25:53.007442 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.007390 2556 generic.go:356] "Generic (PLEG): container finished" podID="d96687e2-3ff2-4347-81a2-2033ba3a5610" containerID="821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed" exitCode=0 Jul 09 14:25:53.007829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.007460 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586f64b4d6-8d84g" Jul 09 14:25:53.007829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.007470 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586f64b4d6-8d84g" event={"ID":"d96687e2-3ff2-4347-81a2-2033ba3a5610","Type":"ContainerDied","Data":"821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed"} Jul 09 14:25:53.007829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.007524 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586f64b4d6-8d84g" event={"ID":"d96687e2-3ff2-4347-81a2-2033ba3a5610","Type":"ContainerDied","Data":"8dce125c3431f59c6a1e983adf890533f3a6a5613f0bde02f9ea8689b6f37f5c"} Jul 09 14:25:53.007829 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.007546 2556 scope.go:117] "RemoveContainer" containerID="821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed" Jul 09 14:25:53.016436 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.016405 2556 scope.go:117] "RemoveContainer" containerID="821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed" Jul 09 14:25:53.016694 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:25:53.016678 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed\": container with ID starting with 821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed not found: ID does not exist" containerID="821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed" Jul 09 14:25:53.016731 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.016704 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed"} err="failed to get container status \"821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed\": rpc error: code = NotFound desc = could not find container \"821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed\": container with ID starting with 821f2de62250a602858d7b54e916edbfc01df1b2a2acc6cef72ea7c523aa33ed not found: ID does not exist" Jul 09 14:25:53.030306 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.030284 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586f64b4d6-8d84g"] Jul 09 14:25:53.034900 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.034875 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-586f64b4d6-8d84g"] Jul 09 14:25:53.849444 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.849383 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:25:53.985979 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.985945 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586f64b4d6-w7w5d"] Jul 09 14:25:53.986465 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.986445 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96687e2-3ff2-4347-81a2-2033ba3a5610" containerName="manager" Jul 09 14:25:53.986465 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.986463 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96687e2-3ff2-4347-81a2-2033ba3a5610" containerName="manager" Jul 09 14:25:53.986609 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.986528 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96687e2-3ff2-4347-81a2-2033ba3a5610" containerName="manager" Jul 09 14:25:53.990143 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.990128 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:53.992693 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.992672 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 14:25:53.993625 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.993606 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 14:25:53.993730 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.993625 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ljgzz\"" Jul 09 14:25:53.999476 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:53.999455 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586f64b4d6-w7w5d"] Jul 09 14:25:54.052185 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.052152 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872zt\" (UniqueName: \"kubernetes.io/projected/b1113875-ac63-4c71-9caa-38c1b7f89a21-kube-api-access-872zt\") pod \"maas-controller-586f64b4d6-w7w5d\" (UID: \"b1113875-ac63-4c71-9caa-38c1b7f89a21\") " pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:54.052553 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.052192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1113875-ac63-4c71-9caa-38c1b7f89a21-webhook-cert\") pod \"maas-controller-586f64b4d6-w7w5d\" (UID: \"b1113875-ac63-4c71-9caa-38c1b7f89a21\") " pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:54.080396 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.076618 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96687e2-3ff2-4347-81a2-2033ba3a5610" path="/var/lib/kubelet/pods/d96687e2-3ff2-4347-81a2-2033ba3a5610/volumes" Jul 09 14:25:54.152901 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.152840 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872zt\" (UniqueName: \"kubernetes.io/projected/b1113875-ac63-4c71-9caa-38c1b7f89a21-kube-api-access-872zt\") pod \"maas-controller-586f64b4d6-w7w5d\" (UID: \"b1113875-ac63-4c71-9caa-38c1b7f89a21\") " pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:54.152901 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.152874 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1113875-ac63-4c71-9caa-38c1b7f89a21-webhook-cert\") pod \"maas-controller-586f64b4d6-w7w5d\" (UID: \"b1113875-ac63-4c71-9caa-38c1b7f89a21\") " pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:54.155227 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.155210 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1113875-ac63-4c71-9caa-38c1b7f89a21-webhook-cert\") pod \"maas-controller-586f64b4d6-w7w5d\" (UID: \"b1113875-ac63-4c71-9caa-38c1b7f89a21\") " pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:54.160561 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.160542 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-872zt\" (UniqueName: \"kubernetes.io/projected/b1113875-ac63-4c71-9caa-38c1b7f89a21-kube-api-access-872zt\") pod \"maas-controller-586f64b4d6-w7w5d\" (UID: \"b1113875-ac63-4c71-9caa-38c1b7f89a21\") " pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:54.301379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.301347 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:54.431515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.431491 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586f64b4d6-w7w5d"] Jul 09 14:25:54.433961 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:25:54.433933 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1113875_ac63_4c71_9caa_38c1b7f89a21.slice/crio-94706ea874434f19ffdd6039c957abfea5d9af1dec371e6548cf70cfddeadfb5 WatchSource:0}: Error finding container 94706ea874434f19ffdd6039c957abfea5d9af1dec371e6548cf70cfddeadfb5: Status 404 returned error can't find the container with id 94706ea874434f19ffdd6039c957abfea5d9af1dec371e6548cf70cfddeadfb5 Jul 09 14:25:54.435315 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:54.435293 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:25:55.019443 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:55.019385 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586f64b4d6-w7w5d" event={"ID":"b1113875-ac63-4c71-9caa-38c1b7f89a21","Type":"ContainerStarted","Data":"94706ea874434f19ffdd6039c957abfea5d9af1dec371e6548cf70cfddeadfb5"} Jul 09 14:25:56.025598 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:56.025560 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586f64b4d6-w7w5d" event={"ID":"b1113875-ac63-4c71-9caa-38c1b7f89a21","Type":"ContainerStarted","Data":"ca8b3dc2d0953787f1c68b0ae8f85ef03056ce65fe1f22e51d4dd3675632c949"} Jul 09 14:25:56.026008 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:56.025637 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:25:56.045903 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:25:56.045853 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-586f64b4d6-w7w5d" podStartSLOduration=2.091271065 podStartE2EDuration="3.045837494s" totalImagesPullingTime="954.566429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:25:56.044794887 +0000 UTC m=+676.649309587" watchObservedRunningTime="2026-07-09 14:25:56.045837494 +0000 UTC m=+676.650352194" Jul 09 14:26:07.036837 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:07.036798 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586f64b4d6-w7w5d" Jul 09 14:26:38.477992 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:38.477908 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:38.577371 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:38.577338 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:38.675595 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:38.675563 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:44.940801 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:44.940768 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:45.107710 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:45.107664 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:45.270189 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:45.270101 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:45.403694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:45.403664 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:51.916605 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:51.916577 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:26:52.469579 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:26:52.469550 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:28:05.568870 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:28:05.568838 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:29:39.915763 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:29:39.915683 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:29:39.918105 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:29:39.918071 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:30:42.435415 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:30:42.435368 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:31:53.140141 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:31:53.140100 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:32:52.630881 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:32:52.630794 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:34:39.941618 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:34:39.941591 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:34:39.945481 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:34:39.945460 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:35:15.870234 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:35:15.870195 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:35:15.929859 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:35:15.929828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:35:36.881025 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:35:36.880940 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:35:36.942639 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:35:36.942609 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:32.809018 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:32.808985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:35.968175 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:35.968145 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:39.429983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:39.429947 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:39.519824 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:39.519792 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:39.685954 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:39.685889 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:39.900955 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:39.900926 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:40.870513 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:40.870473 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:40.968417 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:40.968385 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:41.077576 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:41.077549 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:41.382009 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:41.381978 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:41.503778 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:41.503747 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:41.613441 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:41.613392 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:43.724221 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:43.724184 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:45.893237 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:45.893204 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:46.088373 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:46.088345 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:46.165835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:46.165756 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:46.265768 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:46.265735 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:48.330159 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:48.330129 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:50.536557 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:50.536524 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:52.203887 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:52.203860 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:52.309153 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:52.309124 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:52.474789 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:52.474714 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:52.567306 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:52.567272 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:53.630093 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:53.630055 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:53.730529 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:53.730497 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:53.926658 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:53.926583 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9"] Jul 09 14:37:53.929747 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:53.929731 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:53.932432 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:53.932397 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-899fd65b-serving-cert\"" Jul 09 14:37:53.955575 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:53.955555 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9"] Jul 09 14:37:53.996912 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:53.996886 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:54.024286 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.024264 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611b24c-4906-4974-8a4d-8d008eef9113-maas-api-tls\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.024372 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.024296 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611b24c-4906-4974-8a4d-8d008eef9113-openshift-service-ca\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.024372 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.024320 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjmc\" (UniqueName: \"kubernetes.io/projected/3611b24c-4906-4974-8a4d-8d008eef9113-kube-api-access-8vjmc\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.125586 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.125556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611b24c-4906-4974-8a4d-8d008eef9113-maas-api-tls\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.125711 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.125599 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611b24c-4906-4974-8a4d-8d008eef9113-openshift-service-ca\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.125711 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.125626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjmc\" (UniqueName: \"kubernetes.io/projected/3611b24c-4906-4974-8a4d-8d008eef9113-kube-api-access-8vjmc\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.126159 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.126136 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611b24c-4906-4974-8a4d-8d008eef9113-openshift-service-ca\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.128075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.128050 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611b24c-4906-4974-8a4d-8d008eef9113-maas-api-tls\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.143898 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.143869 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjmc\" (UniqueName: \"kubernetes.io/projected/3611b24c-4906-4974-8a4d-8d008eef9113-kube-api-access-8vjmc\") pod \"maas-api-e2e-ait-899fd65b-74dc44598-jnzl9\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.160403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.160379 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:54.240186 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.240134 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:54.389218 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.389191 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9"] Jul 09 14:37:54.392037 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:37:54.392005 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3611b24c_4906_4974_8a4d_8d008eef9113.slice/crio-fff50c40f072ab5ae8b48e8d3827ef74d94679e085266212987417be9c89da35 WatchSource:0}: Error finding container fff50c40f072ab5ae8b48e8d3827ef74d94679e085266212987417be9c89da35: Status 404 returned error can't find the container with id fff50c40f072ab5ae8b48e8d3827ef74d94679e085266212987417be9c89da35 Jul 09 14:37:54.393365 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.393347 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:37:54.852731 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:54.852704 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:55.051530 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:55.051504 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:55.283010 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:55.282929 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" event={"ID":"3611b24c-4906-4974-8a4d-8d008eef9113","Type":"ContainerStarted","Data":"fff50c40f072ab5ae8b48e8d3827ef74d94679e085266212987417be9c89da35"} Jul 09 14:37:56.251693 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:56.251659 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:56.288887 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:56.288853 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" event={"ID":"3611b24c-4906-4974-8a4d-8d008eef9113","Type":"ContainerStarted","Data":"420890e3c81f4176eac137467ee92fe096e94bc4d8f4d2d3d429d61c541e7938"} Jul 09 14:37:56.289193 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:56.289090 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:56.312579 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:56.312527 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" podStartSLOduration=2.3453549750000002 podStartE2EDuration="3.312509248s" totalImagesPullingTime="967.154273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:37:56.311870437 +0000 UTC m=+1396.916385139" watchObservedRunningTime="2026-07-09 14:37:56.312509248 +0000 UTC m=+1396.917023950" Jul 09 14:37:58.670592 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:58.670555 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:59.061721 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.061689 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9"] Jul 09 14:37:59.062008 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.061980 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" podUID="3611b24c-4906-4974-8a4d-8d008eef9113" containerName="maas-api" containerID="cri-o://420890e3c81f4176eac137467ee92fe096e94bc4d8f4d2d3d429d61c541e7938" gracePeriod=30 Jul 09 14:37:59.114807 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.114776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:37:59.303627 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.303599 2556 generic.go:356] "Generic (PLEG): container finished" podID="3611b24c-4906-4974-8a4d-8d008eef9113" containerID="420890e3c81f4176eac137467ee92fe096e94bc4d8f4d2d3d429d61c541e7938" exitCode=0 Jul 09 14:37:59.303762 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.303675 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" event={"ID":"3611b24c-4906-4974-8a4d-8d008eef9113","Type":"ContainerDied","Data":"420890e3c81f4176eac137467ee92fe096e94bc4d8f4d2d3d429d61c541e7938"} Jul 09 14:37:59.303762 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.303725 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" event={"ID":"3611b24c-4906-4974-8a4d-8d008eef9113","Type":"ContainerDied","Data":"fff50c40f072ab5ae8b48e8d3827ef74d94679e085266212987417be9c89da35"} Jul 09 14:37:59.303762 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.303742 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff50c40f072ab5ae8b48e8d3827ef74d94679e085266212987417be9c89da35" Jul 09 14:37:59.308613 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.308594 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:37:59.474028 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.473952 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611b24c-4906-4974-8a4d-8d008eef9113-maas-api-tls\") pod \"3611b24c-4906-4974-8a4d-8d008eef9113\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " Jul 09 14:37:59.474028 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.473984 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vjmc\" (UniqueName: \"kubernetes.io/projected/3611b24c-4906-4974-8a4d-8d008eef9113-kube-api-access-8vjmc\") pod \"3611b24c-4906-4974-8a4d-8d008eef9113\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " Jul 09 14:37:59.474209 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.474063 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611b24c-4906-4974-8a4d-8d008eef9113-openshift-service-ca\") pod \"3611b24c-4906-4974-8a4d-8d008eef9113\" (UID: \"3611b24c-4906-4974-8a4d-8d008eef9113\") " Jul 09 14:37:59.474507 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.474479 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3611b24c-4906-4974-8a4d-8d008eef9113-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3611b24c-4906-4974-8a4d-8d008eef9113" (UID: "3611b24c-4906-4974-8a4d-8d008eef9113"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:37:59.476179 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.476159 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3611b24c-4906-4974-8a4d-8d008eef9113-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3611b24c-4906-4974-8a4d-8d008eef9113" (UID: "3611b24c-4906-4974-8a4d-8d008eef9113"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:37:59.476264 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.476242 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3611b24c-4906-4974-8a4d-8d008eef9113-kube-api-access-8vjmc" (OuterVolumeSpecName: "kube-api-access-8vjmc") pod "3611b24c-4906-4974-8a4d-8d008eef9113" (UID: "3611b24c-4906-4974-8a4d-8d008eef9113"). InnerVolumeSpecName "kube-api-access-8vjmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:37:59.574743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.574714 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3611b24c-4906-4974-8a4d-8d008eef9113-maas-api-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:37:59.574743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.574740 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjmc\" (UniqueName: \"kubernetes.io/projected/3611b24c-4906-4974-8a4d-8d008eef9113-kube-api-access-8vjmc\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:37:59.574743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:37:59.574750 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3611b24c-4906-4974-8a4d-8d008eef9113-openshift-service-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:38:00.307838 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:00.307811 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9" Jul 09 14:38:00.323480 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:00.323445 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9"] Jul 09 14:38:00.327563 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:00.327540 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-899fd65b-74dc44598-jnzl9"] Jul 09 14:38:02.075830 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:02.075794 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3611b24c-4906-4974-8a4d-8d008eef9113" path="/var/lib/kubelet/pods/3611b24c-4906-4974-8a4d-8d008eef9113/volumes" Jul 09 14:38:04.940329 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:04.940293 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:05.123749 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:05.123701 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:06.798357 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:06.798316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:12.245872 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:12.245835 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:12.477708 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:12.477651 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:12.707174 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:12.707141 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:12.873306 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:12.873276 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:13.788563 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.788527 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:13.902372 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.902337 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7"] Jul 09 14:38:13.902749 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.902733 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3611b24c-4906-4974-8a4d-8d008eef9113" containerName="maas-api" Jul 09 14:38:13.902794 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.902751 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3611b24c-4906-4974-8a4d-8d008eef9113" containerName="maas-api" Jul 09 14:38:13.902827 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.902821 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3611b24c-4906-4974-8a4d-8d008eef9113" containerName="maas-api" Jul 09 14:38:13.905461 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.905445 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:13.908880 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.908857 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-270537f6-serving-cert\"" Jul 09 14:38:13.935573 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.935546 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7"] Jul 09 14:38:13.955475 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.955446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:13.996761 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.996733 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:13.996913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.996766 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvwh\" (UniqueName: \"kubernetes.io/projected/47a6fee1-1333-4369-923d-cdc1a76f0206-kube-api-access-rdvwh\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:13.996913 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:13.996798 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a6fee1-1333-4369-923d-cdc1a76f0206-openshift-service-ca\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.067571 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.067543 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:14.097612 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.097587 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.097736 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.097616 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvwh\" (UniqueName: \"kubernetes.io/projected/47a6fee1-1333-4369-923d-cdc1a76f0206-kube-api-access-rdvwh\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.097736 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.097641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a6fee1-1333-4369-923d-cdc1a76f0206-openshift-service-ca\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.097736 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:38:14.097722 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-serving-cert: secret "maas-api-e2e-derive-270537f6-serving-cert" not found Jul 09 14:38:14.097865 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:38:14.097789 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls podName:47a6fee1-1333-4369-923d-cdc1a76f0206 nodeName:}" failed. No retries permitted until 2026-07-09 14:38:14.597773446 +0000 UTC m=+1415.202288129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls") pod "maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" (UID: "47a6fee1-1333-4369-923d-cdc1a76f0206") : secret "maas-api-e2e-derive-270537f6-serving-cert" not found Jul 09 14:38:14.098195 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.098179 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a6fee1-1333-4369-923d-cdc1a76f0206-openshift-service-ca\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.119989 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.119961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvwh\" (UniqueName: \"kubernetes.io/projected/47a6fee1-1333-4369-923d-cdc1a76f0206-kube-api-access-rdvwh\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.299789 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.299748 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:14.602052 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.602023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.604410 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.604390 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls\") pod \"maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.816359 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.816324 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:14.973689 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:14.973657 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7"] Jul 09 14:38:14.976678 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:38:14.976652 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a6fee1_1333_4369_923d_cdc1a76f0206.slice/crio-e324c77a8d716a49808ee754a4553f85bf91d731107dc2ece8c265446af37118 WatchSource:0}: Error finding container e324c77a8d716a49808ee754a4553f85bf91d731107dc2ece8c265446af37118: Status 404 returned error can't find the container with id e324c77a8d716a49808ee754a4553f85bf91d731107dc2ece8c265446af37118 Jul 09 14:38:15.130406 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:15.130329 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:15.229784 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:15.229748 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:15.375369 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:15.375335 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" event={"ID":"47a6fee1-1333-4369-923d-cdc1a76f0206","Type":"ContainerStarted","Data":"e324c77a8d716a49808ee754a4553f85bf91d731107dc2ece8c265446af37118"} Jul 09 14:38:16.383109 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:16.383068 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" event={"ID":"47a6fee1-1333-4369-923d-cdc1a76f0206","Type":"ContainerStarted","Data":"f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528"} Jul 09 14:38:16.383556 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:16.383151 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:16.415164 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:16.415116 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" podStartSLOduration=2.803066438 podStartE2EDuration="3.415100171s" totalImagesPullingTime="612.033733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:38:16.413175988 +0000 UTC m=+1417.017690689" watchObservedRunningTime="2026-07-09 14:38:16.415100171 +0000 UTC m=+1417.019614868" Jul 09 14:38:16.442605 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:16.442575 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:18.120201 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.120176 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:18.304866 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.304839 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:18.483722 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.483634 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:18.495735 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.495700 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7"] Jul 09 14:38:18.496018 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.495978 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" podUID="47a6fee1-1333-4369-923d-cdc1a76f0206" containerName="maas-api" containerID="cri-o://f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528" gracePeriod=30 Jul 09 14:38:18.579576 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.579550 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:18.758958 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.758934 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:18.837299 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.837268 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a6fee1-1333-4369-923d-cdc1a76f0206-openshift-service-ca\") pod \"47a6fee1-1333-4369-923d-cdc1a76f0206\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " Jul 09 14:38:18.837299 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.837301 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls\") pod \"47a6fee1-1333-4369-923d-cdc1a76f0206\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " Jul 09 14:38:18.837522 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.837353 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdvwh\" (UniqueName: \"kubernetes.io/projected/47a6fee1-1333-4369-923d-cdc1a76f0206-kube-api-access-rdvwh\") pod \"47a6fee1-1333-4369-923d-cdc1a76f0206\" (UID: \"47a6fee1-1333-4369-923d-cdc1a76f0206\") " Jul 09 14:38:18.837697 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.837667 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a6fee1-1333-4369-923d-cdc1a76f0206-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47a6fee1-1333-4369-923d-cdc1a76f0206" (UID: "47a6fee1-1333-4369-923d-cdc1a76f0206"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:38:18.839344 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.839319 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47a6fee1-1333-4369-923d-cdc1a76f0206" (UID: "47a6fee1-1333-4369-923d-cdc1a76f0206"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:38:18.839461 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.839366 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a6fee1-1333-4369-923d-cdc1a76f0206-kube-api-access-rdvwh" (OuterVolumeSpecName: "kube-api-access-rdvwh") pod "47a6fee1-1333-4369-923d-cdc1a76f0206" (UID: "47a6fee1-1333-4369-923d-cdc1a76f0206"). InnerVolumeSpecName "kube-api-access-rdvwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:38:18.938168 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.938134 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a6fee1-1333-4369-923d-cdc1a76f0206-openshift-service-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:38:18.938168 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.938163 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a6fee1-1333-4369-923d-cdc1a76f0206-maas-api-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:38:18.938168 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:18.938176 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvwh\" (UniqueName: \"kubernetes.io/projected/47a6fee1-1333-4369-923d-cdc1a76f0206-kube-api-access-rdvwh\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:38:19.399607 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.399573 2556 generic.go:356] "Generic (PLEG): container finished" podID="47a6fee1-1333-4369-923d-cdc1a76f0206" containerID="f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528" exitCode=0 Jul 09 14:38:19.400051 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.399612 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" Jul 09 14:38:19.400051 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.399641 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" event={"ID":"47a6fee1-1333-4369-923d-cdc1a76f0206","Type":"ContainerDied","Data":"f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528"} Jul 09 14:38:19.400051 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.399696 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7" event={"ID":"47a6fee1-1333-4369-923d-cdc1a76f0206","Type":"ContainerDied","Data":"e324c77a8d716a49808ee754a4553f85bf91d731107dc2ece8c265446af37118"} Jul 09 14:38:19.400051 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.399729 2556 scope.go:117] "RemoveContainer" containerID="f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528" Jul 09 14:38:19.408548 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.408532 2556 scope.go:117] "RemoveContainer" containerID="f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528" Jul 09 14:38:19.408802 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:38:19.408784 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528\": container with ID starting with f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528 not found: ID does not exist" containerID="f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528" Jul 09 14:38:19.408861 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.408809 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528"} err="failed to get container status \"f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528\": rpc error: code = NotFound desc = could not find container \"f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528\": container with ID starting with f7847db6c9ce2b213a32d489e259916f0e3a40b22ae834c1a9ffc73d27357528 not found: ID does not exist" Jul 09 14:38:19.423735 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.423714 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7"] Jul 09 14:38:19.430319 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.430299 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-270537f6-5b5d9b78fd-j5hp7"] Jul 09 14:38:19.865815 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:19.865775 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:20.075391 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:20.075356 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47a6fee1-1333-4369-923d-cdc1a76f0206" path="/var/lib/kubelet/pods/47a6fee1-1333-4369-923d-cdc1a76f0206/volumes" Jul 09 14:38:24.445693 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:24.445656 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:24.518919 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:24.518887 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:24.998104 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:24.998073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:25.139350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:25.139326 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:26.537157 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:26.537125 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:32.478537 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:32.478415 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:32.600488 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:32.600446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:32.725397 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:32.725366 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:32.825378 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:32.825319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:33.222039 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:33.221956 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:49.052047 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:49.052007 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:38:59.220178 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:38:59.220140 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:05.925541 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:05.925505 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:06.153868 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:06.153833 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:06.284507 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:06.284364 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:08.652929 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:08.652866 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:18.551687 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:18.551653 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:22.124558 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:22.124526 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:29.436473 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:29.436415 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:29.558296 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:29.558267 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:29.683180 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:29.683148 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:39.967729 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:39.967697 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:39:39.972581 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:39.972559 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:39:40.297585 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:40.297561 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:41.079619 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:41.079585 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:42.533975 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:42.533945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:42.633110 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:42.633077 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:47.690520 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:47.690484 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:47.910603 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:47.910568 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:48.120621 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:48.120594 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:48.989698 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:48.989651 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:49.116439 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:49.116381 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:50.768411 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:50.768373 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:57.471986 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:57.471952 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:57.541853 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:57.541822 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:39:58.194801 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:39:58.194760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:04.601156 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:04.601118 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:04.714692 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:04.714663 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:04.812413 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:04.812385 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:05.091064 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:05.091031 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:05.312896 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:05.312866 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:08.010263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:08.010229 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:08.260081 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:08.260050 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:08.501239 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:08.501212 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:08.929245 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:08.929215 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:09.028739 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:09.028703 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:12.368226 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:12.368194 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:13.025622 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:13.025588 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:13.114836 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:13.114808 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:13.423337 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:13.423306 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:19.427545 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:19.427516 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:19.526821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:19.526790 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:19.830717 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:19.830683 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:20.212629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:20.212547 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:20.307664 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:20.307627 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:26.183115 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:26.183053 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:26.283972 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:26.283942 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:26.458382 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:26.458302 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:26.675561 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:26.675529 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:26.805181 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:26.805151 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:26.896292 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:26.896260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:27.002140 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:27.002111 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:28.022562 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:28.022531 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:28.249878 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:28.249850 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:32.785934 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:32.785901 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:32.895250 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:32.895213 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:32.986702 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:32.986661 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:33.140809 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:33.140773 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:39.117061 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:39.117025 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:39.217533 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:39.217501 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:45.914118 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:45.914073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:46.233383 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:46.233295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:46.425629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:46.425600 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:49.223480 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.223439 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:49.321410 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.321369 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:49.474783 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.474681 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf"] Jul 09 14:40:49.475243 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.475223 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a6fee1-1333-4369-923d-cdc1a76f0206" containerName="maas-api" Jul 09 14:40:49.475300 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.475244 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a6fee1-1333-4369-923d-cdc1a76f0206" containerName="maas-api" Jul 09 14:40:49.475333 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.475302 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a6fee1-1333-4369-923d-cdc1a76f0206" containerName="maas-api" Jul 09 14:40:49.478339 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.478316 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.480943 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.480923 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2a4ea46b-serving-cert\"" Jul 09 14:40:49.490362 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.490330 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf"] Jul 09 14:40:49.520587 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.520560 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:49.546782 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.546760 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ee4505-3379-4ed7-a271-3f832941ea64-maas-api-tls\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.546890 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.546814 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee4505-3379-4ed7-a271-3f832941ea64-openshift-service-ca\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.546940 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.546918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh52v\" (UniqueName: \"kubernetes.io/projected/e7ee4505-3379-4ed7-a271-3f832941ea64-kube-api-access-zh52v\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.644146 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.644115 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:49.648263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.648244 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee4505-3379-4ed7-a271-3f832941ea64-openshift-service-ca\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.648332 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.648288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh52v\" (UniqueName: \"kubernetes.io/projected/e7ee4505-3379-4ed7-a271-3f832941ea64-kube-api-access-zh52v\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.648332 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.648313 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ee4505-3379-4ed7-a271-3f832941ea64-maas-api-tls\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.648875 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.648848 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee4505-3379-4ed7-a271-3f832941ea64-openshift-service-ca\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.650796 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.650774 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ee4505-3379-4ed7-a271-3f832941ea64-maas-api-tls\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.656330 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.656308 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh52v\" (UniqueName: \"kubernetes.io/projected/e7ee4505-3379-4ed7-a271-3f832941ea64-kube-api-access-zh52v\") pod \"maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.792174 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.792099 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:49.932303 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:49.932281 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf"] Jul 09 14:40:49.934989 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:40:49.934954 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ee4505_3379_4ed7_a271_3f832941ea64.slice/crio-25459a0707d673e6cd443cd5cb8ff4bb88a845474273e75d76c5f21b9b3c9385 WatchSource:0}: Error finding container 25459a0707d673e6cd443cd5cb8ff4bb88a845474273e75d76c5f21b9b3c9385: Status 404 returned error can't find the container with id 25459a0707d673e6cd443cd5cb8ff4bb88a845474273e75d76c5f21b9b3c9385 Jul 09 14:40:50.006629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:50.006604 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:50.107411 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:50.107376 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:50.114872 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:50.114848 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" event={"ID":"e7ee4505-3379-4ed7-a271-3f832941ea64","Type":"ContainerStarted","Data":"25459a0707d673e6cd443cd5cb8ff4bb88a845474273e75d76c5f21b9b3c9385"} Jul 09 14:40:51.122150 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:51.122115 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" event={"ID":"e7ee4505-3379-4ed7-a271-3f832941ea64","Type":"ContainerStarted","Data":"142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d"} Jul 09 14:40:51.122662 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:51.122181 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:40:51.140147 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:51.140097 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" podStartSLOduration=1.7822171500000001 podStartE2EDuration="2.140082658s" totalImagesPullingTime="357.865508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:40:51.138927475 +0000 UTC m=+1571.743442179" watchObservedRunningTime="2026-07-09 14:40:51.140082658 +0000 UTC m=+1571.744597357" Jul 09 14:40:53.744213 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:53.744180 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:40:57.132970 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:40:57.132941 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:41:02.679519 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:02.679485 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:16.527613 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:16.527577 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:16.628323 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:16.628290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:21.241340 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:21.241305 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:21.514441 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:21.514345 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:21.767352 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:21.767267 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:21.853098 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:21.853065 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf"] Jul 09 14:41:21.853301 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:21.853279 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" podUID="e7ee4505-3379-4ed7-a271-3f832941ea64" containerName="maas-api" containerID="cri-o://142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d" gracePeriod=30 Jul 09 14:41:21.881570 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:21.881541 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:22.100551 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.100527 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:41:22.139710 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.139684 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh52v\" (UniqueName: \"kubernetes.io/projected/e7ee4505-3379-4ed7-a271-3f832941ea64-kube-api-access-zh52v\") pod \"e7ee4505-3379-4ed7-a271-3f832941ea64\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " Jul 09 14:41:22.139846 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.139725 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ee4505-3379-4ed7-a271-3f832941ea64-maas-api-tls\") pod \"e7ee4505-3379-4ed7-a271-3f832941ea64\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " Jul 09 14:41:22.139846 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.139817 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee4505-3379-4ed7-a271-3f832941ea64-openshift-service-ca\") pod \"e7ee4505-3379-4ed7-a271-3f832941ea64\" (UID: \"e7ee4505-3379-4ed7-a271-3f832941ea64\") " Jul 09 14:41:22.140215 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.140186 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ee4505-3379-4ed7-a271-3f832941ea64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7ee4505-3379-4ed7-a271-3f832941ea64" (UID: "e7ee4505-3379-4ed7-a271-3f832941ea64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:41:22.141822 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.141794 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ee4505-3379-4ed7-a271-3f832941ea64-kube-api-access-zh52v" (OuterVolumeSpecName: "kube-api-access-zh52v") pod "e7ee4505-3379-4ed7-a271-3f832941ea64" (UID: "e7ee4505-3379-4ed7-a271-3f832941ea64"). InnerVolumeSpecName "kube-api-access-zh52v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:41:22.141914 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.141890 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ee4505-3379-4ed7-a271-3f832941ea64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7ee4505-3379-4ed7-a271-3f832941ea64" (UID: "e7ee4505-3379-4ed7-a271-3f832941ea64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:41:22.241011 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.240984 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh52v\" (UniqueName: \"kubernetes.io/projected/e7ee4505-3379-4ed7-a271-3f832941ea64-kube-api-access-zh52v\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:41:22.241011 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.241010 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ee4505-3379-4ed7-a271-3f832941ea64-maas-api-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:41:22.241183 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.241020 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ee4505-3379-4ed7-a271-3f832941ea64-openshift-service-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:41:22.256863 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.256832 2556 generic.go:356] "Generic (PLEG): container finished" podID="e7ee4505-3379-4ed7-a271-3f832941ea64" containerID="142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d" exitCode=0 Jul 09 14:41:22.257263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.256882 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" Jul 09 14:41:22.257263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.256922 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" event={"ID":"e7ee4505-3379-4ed7-a271-3f832941ea64","Type":"ContainerDied","Data":"142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d"} Jul 09 14:41:22.257263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.256990 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf" event={"ID":"e7ee4505-3379-4ed7-a271-3f832941ea64","Type":"ContainerDied","Data":"25459a0707d673e6cd443cd5cb8ff4bb88a845474273e75d76c5f21b9b3c9385"} Jul 09 14:41:22.257263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.257018 2556 scope.go:117] "RemoveContainer" containerID="142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d" Jul 09 14:41:22.266332 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.266318 2556 scope.go:117] "RemoveContainer" containerID="142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d" Jul 09 14:41:22.266612 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:41:22.266596 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d\": container with ID starting with 142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d not found: ID does not exist" containerID="142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d" Jul 09 14:41:22.266656 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.266620 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d"} err="failed to get container status \"142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d\": rpc error: code = NotFound desc = could not find container \"142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d\": container with ID starting with 142605fd7c017c37af6d8f162eff602c26b77a5248cbf199f3cdc5464794723d not found: ID does not exist" Jul 09 14:41:22.277825 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.277774 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf"] Jul 09 14:41:22.283931 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:22.283906 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a4ea46b-6f6466dc7f-f4zbf"] Jul 09 14:41:24.075882 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:24.075849 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7ee4505-3379-4ed7-a271-3f832941ea64" path="/var/lib/kubelet/pods/e7ee4505-3379-4ed7-a271-3f832941ea64/volumes" Jul 09 14:41:27.121012 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:27.120979 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:41:32.378332 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:41:32.378249 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:08.558987 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:08.558954 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:09.388908 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:09.388878 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:15.319610 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:15.319580 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:15.400147 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:15.400110 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:15.614102 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:15.614019 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:15.848997 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:15.848965 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:16.939662 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:16.939627 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:17.063076 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:17.063037 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:17.880016 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:17.879973 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:18.021130 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:18.021098 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:19.450079 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:19.450047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:25.394553 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:25.394520 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:25.723921 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:25.723846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:26.104614 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:26.104580 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:31.851154 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:31.851117 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:32.021199 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:32.021166 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:32.180104 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:32.180023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:42.649936 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:42.649902 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:42:44.467539 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:42:44.467493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:03.811190 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:03.811154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:09.660653 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:09.660616 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:09.760805 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:09.760771 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:10.026619 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:10.026516 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:10.310316 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:10.310282 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:10.570648 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:10.570566 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:12.815236 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.815201 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:12.886307 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.886279 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:12.890760 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.890729 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg"] Jul 09 14:43:12.891081 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.891068 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ee4505-3379-4ed7-a271-3f832941ea64" containerName="maas-api" Jul 09 14:43:12.891162 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.891082 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ee4505-3379-4ed7-a271-3f832941ea64" containerName="maas-api" Jul 09 14:43:12.891162 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.891155 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ee4505-3379-4ed7-a271-3f832941ea64" containerName="maas-api" Jul 09 14:43:12.894196 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.894175 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:12.896345 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.896327 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d2daa1-serving-cert\"" Jul 09 14:43:12.904128 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.904102 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg"] Jul 09 14:43:12.962028 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.961989 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:12.999157 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.999129 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3eceb5-f17b-49ee-9002-baed79901b0f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:12.999273 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.999260 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:12.999314 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:12.999286 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcggt\" (UniqueName: \"kubernetes.io/projected/4e3eceb5-f17b-49ee-9002-baed79901b0f-kube-api-access-jcggt\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.099921 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.099834 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcggt\" (UniqueName: \"kubernetes.io/projected/4e3eceb5-f17b-49ee-9002-baed79901b0f-kube-api-access-jcggt\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.099921 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.099875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3eceb5-f17b-49ee-9002-baed79901b0f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.100152 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.099998 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.100152 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:43:13.100104 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-serving-cert: secret "maas-api-e2e-inf-a-d2daa1-serving-cert" not found Jul 09 14:43:13.100274 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:43:13.100169 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls podName:4e3eceb5-f17b-49ee-9002-baed79901b0f nodeName:}" failed. No retries permitted until 2026-07-09 14:43:13.600148398 +0000 UTC m=+1714.204663094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls") pod "maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" (UID: "4e3eceb5-f17b-49ee-9002-baed79901b0f") : secret "maas-api-e2e-inf-a-d2daa1-serving-cert" not found Jul 09 14:43:13.100585 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.100564 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3eceb5-f17b-49ee-9002-baed79901b0f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.108442 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.108408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcggt\" (UniqueName: \"kubernetes.io/projected/4e3eceb5-f17b-49ee-9002-baed79901b0f-kube-api-access-jcggt\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.220260 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.220226 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:13.535991 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.535891 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:13.604521 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.604490 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.606860 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.606837 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.635591 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.635566 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:13.805844 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.805800 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:13.948875 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.948849 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg"] Jul 09 14:43:13.951177 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:43:13.951151 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3eceb5_f17b_49ee_9002_baed79901b0f.slice/crio-eba8b93c6efe195de64c8776dac60fee6f1e65a44cf2b96e5e5da4d0987c2fb4 WatchSource:0}: Error finding container eba8b93c6efe195de64c8776dac60fee6f1e65a44cf2b96e5e5da4d0987c2fb4: Status 404 returned error can't find the container with id eba8b93c6efe195de64c8776dac60fee6f1e65a44cf2b96e5e5da4d0987c2fb4 Jul 09 14:43:13.952561 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:13.952541 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:43:14.760876 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:14.760843 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" event={"ID":"4e3eceb5-f17b-49ee-9002-baed79901b0f","Type":"ContainerStarted","Data":"f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84"} Jul 09 14:43:14.761046 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:14.760894 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" event={"ID":"4e3eceb5-f17b-49ee-9002-baed79901b0f","Type":"ContainerStarted","Data":"eba8b93c6efe195de64c8776dac60fee6f1e65a44cf2b96e5e5da4d0987c2fb4"} Jul 09 14:43:14.761046 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:14.760944 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:14.781169 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:14.781122 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" podStartSLOduration=2.172207045 podStartE2EDuration="2.7811071s" totalImagesPullingTime="608.900055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:43:14.779100211 +0000 UTC m=+1715.383614912" watchObservedRunningTime="2026-07-09 14:43:14.7811071 +0000 UTC m=+1715.385621800" Jul 09 14:43:17.300597 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:17.300565 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:17.624356 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:17.624281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:17.925629 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:17.925553 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:18.115000 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:18.114970 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:20.770794 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:20.770762 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:43:20.954462 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:20.954413 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:21.046573 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:21.046493 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:21.114694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:21.114653 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:21.366205 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:21.366173 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:21.538563 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:21.538532 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:21.630698 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:21.630613 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:25.435594 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:25.435556 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:34.258989 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:34.258931 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:34.359811 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:34.359783 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:39.066135 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.066096 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s"] Jul 09 14:43:39.069621 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.069595 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.072848 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.072818 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d2daa1\"/\"openshift-service-ca.crt\"" Jul 09 14:43:39.072848 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.072842 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d2daa1\"/\"default-dockercfg-wbncx\"" Jul 09 14:43:39.073044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.072885 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d2daa1\"/\"kube-root-ca.crt\"" Jul 09 14:43:39.073044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.072890 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d2daa1\"/\"test-model-d2daa1-kserve-self-signed-certs\"" Jul 09 14:43:39.078212 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.078191 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s"] Jul 09 14:43:39.094828 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.094792 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:39.131198 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.131174 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tmp-dir\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.131295 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.131239 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-home\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.131295 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.131264 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-model-cache\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.131367 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.131300 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tls-certs\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.131367 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.131338 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m987\" (UniqueName: \"kubernetes.io/projected/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-kube-api-access-8m987\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.131458 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.131367 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-dshm\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.232491 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tmp-dir\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.232667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232545 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-home\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.232667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232562 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-model-cache\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.232667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232579 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tls-certs\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.232667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m987\" (UniqueName: \"kubernetes.io/projected/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-kube-api-access-8m987\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.232667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-dshm\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.232985 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232963 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tmp-dir\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.233053 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.232989 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-model-cache\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.233053 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.233028 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-home\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.234743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.234721 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-dshm\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.234867 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.234851 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tls-certs\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.240049 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.240024 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m987\" (UniqueName: \"kubernetes.io/projected/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-kube-api-access-8m987\") pod \"test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.379983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.379896 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:39.515530 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.515490 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s"] Jul 09 14:43:39.518734 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:43:39.518705 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a314fc6_16b9_4864_ad04_ea3f346aaa2f.slice/crio-c3839f267db91fc00e5a4ee45ff14c51b09c28fba3d3a6dda44d2aa1e048539d WatchSource:0}: Error finding container c3839f267db91fc00e5a4ee45ff14c51b09c28fba3d3a6dda44d2aa1e048539d: Status 404 returned error can't find the container with id c3839f267db91fc00e5a4ee45ff14c51b09c28fba3d3a6dda44d2aa1e048539d Jul 09 14:43:39.882043 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.882013 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" event={"ID":"7a314fc6-16b9-4864-ad04-ea3f346aaa2f","Type":"ContainerStarted","Data":"92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866"} Jul 09 14:43:39.882216 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.882050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" event={"ID":"7a314fc6-16b9-4864-ad04-ea3f346aaa2f","Type":"ContainerStarted","Data":"c3839f267db91fc00e5a4ee45ff14c51b09c28fba3d3a6dda44d2aa1e048539d"} Jul 09 14:43:39.900568 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:39.900523 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" podStartSLOduration=0.900506793 podStartE2EDuration="900.506793ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:43:39.898854463 +0000 UTC m=+1740.503369185" watchObservedRunningTime="2026-07-09 14:43:39.900506793 +0000 UTC m=+1740.505021492" Jul 09 14:43:43.460597 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:43.460567 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:43.560950 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:43.560916 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:43:49.380472 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:49.380412 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:49.380949 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:49.380497 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:49.392774 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:49.392748 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:49.947968 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:49.947933 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:43:53.891131 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:43:53.891095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:18.652726 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:18.652693 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s"] Jul 09 14:44:18.675689 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:18.675662 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:19.577762 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:19.577726 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:19.744395 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:19.744363 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:20.172624 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.172591 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:20.270048 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.270013 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:20.377073 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.377039 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:20.475017 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.474928 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg"] Jul 09 14:44:20.475255 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.475210 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" podUID="4e3eceb5-f17b-49ee-9002-baed79901b0f" containerName="maas-api" containerID="cri-o://f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84" gracePeriod=30 Jul 09 14:44:20.505311 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.505281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:20.731014 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.730962 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:44:20.817325 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.817292 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcggt\" (UniqueName: \"kubernetes.io/projected/4e3eceb5-f17b-49ee-9002-baed79901b0f-kube-api-access-jcggt\") pod \"4e3eceb5-f17b-49ee-9002-baed79901b0f\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " Jul 09 14:44:20.817770 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.817355 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls\") pod \"4e3eceb5-f17b-49ee-9002-baed79901b0f\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " Jul 09 14:44:20.817770 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.817379 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3eceb5-f17b-49ee-9002-baed79901b0f-openshift-service-ca\") pod \"4e3eceb5-f17b-49ee-9002-baed79901b0f\" (UID: \"4e3eceb5-f17b-49ee-9002-baed79901b0f\") " Jul 09 14:44:20.817770 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.817760 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3eceb5-f17b-49ee-9002-baed79901b0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e3eceb5-f17b-49ee-9002-baed79901b0f" (UID: "4e3eceb5-f17b-49ee-9002-baed79901b0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:44:20.819576 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.819554 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3eceb5-f17b-49ee-9002-baed79901b0f-kube-api-access-jcggt" (OuterVolumeSpecName: "kube-api-access-jcggt") pod "4e3eceb5-f17b-49ee-9002-baed79901b0f" (UID: "4e3eceb5-f17b-49ee-9002-baed79901b0f"). InnerVolumeSpecName "kube-api-access-jcggt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:44:20.819717 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.819694 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e3eceb5-f17b-49ee-9002-baed79901b0f" (UID: "4e3eceb5-f17b-49ee-9002-baed79901b0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:44:20.918208 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.918173 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3eceb5-f17b-49ee-9002-baed79901b0f-maas-api-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:20.918208 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.918202 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3eceb5-f17b-49ee-9002-baed79901b0f-openshift-service-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:20.918399 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:20.918216 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcggt\" (UniqueName: \"kubernetes.io/projected/4e3eceb5-f17b-49ee-9002-baed79901b0f-kube-api-access-jcggt\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:21.079260 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.079223 2556 generic.go:356] "Generic (PLEG): container finished" podID="4e3eceb5-f17b-49ee-9002-baed79901b0f" containerID="f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84" exitCode=0 Jul 09 14:44:21.079440 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.079267 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" Jul 09 14:44:21.079440 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.079316 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" event={"ID":"4e3eceb5-f17b-49ee-9002-baed79901b0f","Type":"ContainerDied","Data":"f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84"} Jul 09 14:44:21.079440 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.079371 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg" event={"ID":"4e3eceb5-f17b-49ee-9002-baed79901b0f","Type":"ContainerDied","Data":"eba8b93c6efe195de64c8776dac60fee6f1e65a44cf2b96e5e5da4d0987c2fb4"} Jul 09 14:44:21.079440 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.079403 2556 scope.go:117] "RemoveContainer" containerID="f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84" Jul 09 14:44:21.088471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.088451 2556 scope.go:117] "RemoveContainer" containerID="f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84" Jul 09 14:44:21.088735 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:44:21.088716 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84\": container with ID starting with f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84 not found: ID does not exist" containerID="f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84" Jul 09 14:44:21.088803 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.088742 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84"} err="failed to get container status \"f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84\": rpc error: code = NotFound desc = could not find container \"f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84\": container with ID starting with f68e5fe5b1f1de57902e30ec17cc5e91b58986342950d6b8b6a58f740d2ead84 not found: ID does not exist" Jul 09 14:44:21.104338 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.104313 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg"] Jul 09 14:44:21.109827 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:21.109805 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2daa1-594646d4f5-5glbg"] Jul 09 14:44:22.075938 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:22.075909 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e3eceb5-f17b-49ee-9002-baed79901b0f" path="/var/lib/kubelet/pods/4e3eceb5-f17b-49ee-9002-baed79901b0f/volumes" Jul 09 14:44:25.544938 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:25.544891 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:33.327654 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:33.327580 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:33.427880 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:33.427846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:44:33.674507 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:33.674409 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" podUID="7a314fc6-16b9-4864-ad04-ea3f346aaa2f" containerName="main" containerID="cri-o://92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866" gracePeriod=45 Jul 09 14:44:33.933602 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:33.933549 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:44:34.031554 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031525 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m987\" (UniqueName: \"kubernetes.io/projected/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-kube-api-access-8m987\") pod \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " Jul 09 14:44:34.031719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031571 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tmp-dir\") pod \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " Jul 09 14:44:34.031719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031593 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-home\") pod \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " Jul 09 14:44:34.031719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031636 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-dshm\") pod \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " Jul 09 14:44:34.031719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031680 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tls-certs\") pod \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " Jul 09 14:44:34.031719 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031706 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-model-cache\") pod \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\" (UID: \"7a314fc6-16b9-4864-ad04-ea3f346aaa2f\") " Jul 09 14:44:34.031936 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031906 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-home" (OuterVolumeSpecName: "home") pod "7a314fc6-16b9-4864-ad04-ea3f346aaa2f" (UID: "7a314fc6-16b9-4864-ad04-ea3f346aaa2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:44:34.031936 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.031919 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a314fc6-16b9-4864-ad04-ea3f346aaa2f" (UID: "7a314fc6-16b9-4864-ad04-ea3f346aaa2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:44:34.032085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.032061 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tmp-dir\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:34.032085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.032084 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-home\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:34.032085 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.032073 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a314fc6-16b9-4864-ad04-ea3f346aaa2f" (UID: "7a314fc6-16b9-4864-ad04-ea3f346aaa2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:44:34.033863 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.033835 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-dshm" (OuterVolumeSpecName: "dshm") pod "7a314fc6-16b9-4864-ad04-ea3f346aaa2f" (UID: "7a314fc6-16b9-4864-ad04-ea3f346aaa2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:44:34.033863 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.033854 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-kube-api-access-8m987" (OuterVolumeSpecName: "kube-api-access-8m987") pod "7a314fc6-16b9-4864-ad04-ea3f346aaa2f" (UID: "7a314fc6-16b9-4864-ad04-ea3f346aaa2f"). InnerVolumeSpecName "kube-api-access-8m987". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:44:34.033863 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.033844 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a314fc6-16b9-4864-ad04-ea3f346aaa2f" (UID: "7a314fc6-16b9-4864-ad04-ea3f346aaa2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:44:34.133583 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.133554 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m987\" (UniqueName: \"kubernetes.io/projected/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-kube-api-access-8m987\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:34.133583 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.133584 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-dshm\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:34.133799 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.133597 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-tls-certs\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:34.133799 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.133609 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a314fc6-16b9-4864-ad04-ea3f346aaa2f-model-cache\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:44:34.141139 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.141114 2556 generic.go:356] "Generic (PLEG): container finished" podID="7a314fc6-16b9-4864-ad04-ea3f346aaa2f" containerID="92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866" exitCode=0 Jul 09 14:44:34.141266 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.141155 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" Jul 09 14:44:34.141266 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.141197 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" event={"ID":"7a314fc6-16b9-4864-ad04-ea3f346aaa2f","Type":"ContainerDied","Data":"92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866"} Jul 09 14:44:34.141266 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.141241 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s" event={"ID":"7a314fc6-16b9-4864-ad04-ea3f346aaa2f","Type":"ContainerDied","Data":"c3839f267db91fc00e5a4ee45ff14c51b09c28fba3d3a6dda44d2aa1e048539d"} Jul 09 14:44:34.141266 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.141260 2556 scope.go:117] "RemoveContainer" containerID="92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866" Jul 09 14:44:34.149686 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.149668 2556 scope.go:117] "RemoveContainer" containerID="92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866" Jul 09 14:44:34.149959 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:44:34.149942 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866\": container with ID starting with 92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866 not found: ID does not exist" containerID="92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866" Jul 09 14:44:34.150004 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.149969 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866"} err="failed to get container status \"92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866\": rpc error: code = NotFound desc = could not find container \"92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866\": container with ID starting with 92d8071e33bc2df2d1561f9045267f32945f6346525da141e1859680b1dc9866 not found: ID does not exist" Jul 09 14:44:34.156922 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.156901 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s"] Jul 09 14:44:34.161081 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:34.161061 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d2daa1/test-model-d2daa1-kserve-5c5f7ff6c5-hmh6s"] Jul 09 14:44:36.075946 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:36.075912 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a314fc6-16b9-4864-ad04-ea3f346aaa2f" path="/var/lib/kubelet/pods/7a314fc6-16b9-4864-ad04-ea3f346aaa2f/volumes" Jul 09 14:44:39.994462 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:39.994394 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:44:40.000295 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:40.000274 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:44:40.060188 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:44:40.060159 2556 scope.go:117] "RemoveContainer" containerID="420890e3c81f4176eac137467ee92fe096e94bc4d8f4d2d3d429d61c541e7938" Jul 09 14:45:02.729547 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:02.729516 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:02.957538 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:02.957509 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:03.207203 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:03.207167 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:03.369284 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:03.369253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:03.817526 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:03.817492 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:03.979308 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:03.979275 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:04.081470 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:04.081385 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:04.285669 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:04.285639 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:09.622947 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:09.622916 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:09.723727 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:09.723700 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:09.931062 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:09.930984 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:10.171189 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:10.171156 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:10.530553 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:10.530524 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:10.869794 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:10.869767 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:11.806075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:11.806044 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:13.527746 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.527704 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:13.594467 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.594411 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp"] Jul 09 14:45:13.595021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.595001 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a314fc6-16b9-4864-ad04-ea3f346aaa2f" containerName="main" Jul 09 14:45:13.595021 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.595022 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a314fc6-16b9-4864-ad04-ea3f346aaa2f" containerName="main" Jul 09 14:45:13.595189 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.595060 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3eceb5-f17b-49ee-9002-baed79901b0f" containerName="maas-api" Jul 09 14:45:13.595189 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.595070 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3eceb5-f17b-49ee-9002-baed79901b0f" containerName="maas-api" Jul 09 14:45:13.595189 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.595166 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a314fc6-16b9-4864-ad04-ea3f346aaa2f" containerName="main" Jul 09 14:45:13.595189 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.595179 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3eceb5-f17b-49ee-9002-baed79901b0f" containerName="maas-api" Jul 09 14:45:13.598680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.598659 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.601067 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.601049 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-577c63-serving-cert\"" Jul 09 14:45:13.605776 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.605395 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:13.606631 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.606605 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp"] Jul 09 14:45:13.669439 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.669385 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813c361b-e9be-4a20-bb81-7ca890c21e6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.669606 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.669453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.669606 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.669474 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65jbw\" (UniqueName: \"kubernetes.io/projected/813c361b-e9be-4a20-bb81-7ca890c21e6c-kube-api-access-65jbw\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.674524 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.674506 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:13.770015 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.769982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813c361b-e9be-4a20-bb81-7ca890c21e6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.770015 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.770018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.770280 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.770035 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65jbw\" (UniqueName: \"kubernetes.io/projected/813c361b-e9be-4a20-bb81-7ca890c21e6c-kube-api-access-65jbw\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.770280 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:45:13.770151 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-serving-cert: secret "maas-api-e2e-api-a-577c63-serving-cert" not found Jul 09 14:45:13.770280 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:45:13.770227 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls podName:813c361b-e9be-4a20-bb81-7ca890c21e6c nodeName:}" failed. No retries permitted until 2026-07-09 14:45:14.270208677 +0000 UTC m=+1834.874723361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls") pod "maas-api-e2e-api-a-577c63-74769d558b-k9hvp" (UID: "813c361b-e9be-4a20-bb81-7ca890c21e6c") : secret "maas-api-e2e-api-a-577c63-serving-cert" not found Jul 09 14:45:13.770755 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.770732 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813c361b-e9be-4a20-bb81-7ca890c21e6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:13.783374 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:13.783316 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65jbw\" (UniqueName: \"kubernetes.io/projected/813c361b-e9be-4a20-bb81-7ca890c21e6c-kube-api-access-65jbw\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:14.001207 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:14.001174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:14.101504 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:14.101476 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:14.201580 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:14.201544 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:14.274243 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:14.274213 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:14.276660 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:14.276632 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls\") pod \"maas-api-e2e-api-a-577c63-74769d558b-k9hvp\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:14.511660 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:14.511570 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:14.646776 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:14.646747 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp"] Jul 09 14:45:14.649598 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:45:14.649563 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813c361b_e9be_4a20_bb81_7ca890c21e6c.slice/crio-9203037ab228474eb6ec99a10598211f799b911e37703ba08ad5a31f2f353b90 WatchSource:0}: Error finding container 9203037ab228474eb6ec99a10598211f799b911e37703ba08ad5a31f2f353b90: Status 404 returned error can't find the container with id 9203037ab228474eb6ec99a10598211f799b911e37703ba08ad5a31f2f353b90 Jul 09 14:45:15.332679 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:15.332603 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" event={"ID":"813c361b-e9be-4a20-bb81-7ca890c21e6c","Type":"ContainerStarted","Data":"32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d"} Jul 09 14:45:15.332679 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:15.332653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" event={"ID":"813c361b-e9be-4a20-bb81-7ca890c21e6c","Type":"ContainerStarted","Data":"9203037ab228474eb6ec99a10598211f799b911e37703ba08ad5a31f2f353b90"} Jul 09 14:45:15.332858 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:15.332715 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:15.352967 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:15.352921 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" podStartSLOduration=1.930543991 podStartE2EDuration="2.352904643s" totalImagesPullingTime="422.360652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:15.351778188 +0000 UTC m=+1835.956292898" watchObservedRunningTime="2026-07-09 14:45:15.352904643 +0000 UTC m=+1835.957419343" Jul 09 14:45:18.106835 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:18.106804 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:18.596561 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:18.596527 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:18.941376 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:18.941281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:19.194699 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:19.194607 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:21.343129 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:21.343097 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:21.625750 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:21.625668 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:21.733299 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:21.733253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:21.844104 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:21.844070 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:21.973401 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:21.973312 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:22.266011 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:22.265931 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:22.364217 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:22.364187 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:26.121782 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:26.121746 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:34.546044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:34.546010 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:34.815471 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:34.815417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:34.914620 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:34.914587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:35.013936 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.013906 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:35.048821 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.048793 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp"] Jul 09 14:45:35.049242 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.049209 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" podUID="813c361b-e9be-4a20-bb81-7ca890c21e6c" containerName="maas-api" containerID="cri-o://32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d" gracePeriod=30 Jul 09 14:45:35.094306 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.094249 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:35.294825 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.294800 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:35.362454 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.362363 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65jbw\" (UniqueName: \"kubernetes.io/projected/813c361b-e9be-4a20-bb81-7ca890c21e6c-kube-api-access-65jbw\") pod \"813c361b-e9be-4a20-bb81-7ca890c21e6c\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " Jul 09 14:45:35.362454 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.362439 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813c361b-e9be-4a20-bb81-7ca890c21e6c-openshift-service-ca\") pod \"813c361b-e9be-4a20-bb81-7ca890c21e6c\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " Jul 09 14:45:35.362672 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.362480 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls\") pod \"813c361b-e9be-4a20-bb81-7ca890c21e6c\" (UID: \"813c361b-e9be-4a20-bb81-7ca890c21e6c\") " Jul 09 14:45:35.362784 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.362762 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813c361b-e9be-4a20-bb81-7ca890c21e6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "813c361b-e9be-4a20-bb81-7ca890c21e6c" (UID: "813c361b-e9be-4a20-bb81-7ca890c21e6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:45:35.364494 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.364465 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "813c361b-e9be-4a20-bb81-7ca890c21e6c" (UID: "813c361b-e9be-4a20-bb81-7ca890c21e6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:45:35.364494 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.364408 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813c361b-e9be-4a20-bb81-7ca890c21e6c-kube-api-access-65jbw" (OuterVolumeSpecName: "kube-api-access-65jbw") pod "813c361b-e9be-4a20-bb81-7ca890c21e6c" (UID: "813c361b-e9be-4a20-bb81-7ca890c21e6c"). InnerVolumeSpecName "kube-api-access-65jbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:45:35.425178 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.425144 2556 generic.go:356] "Generic (PLEG): container finished" podID="813c361b-e9be-4a20-bb81-7ca890c21e6c" containerID="32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d" exitCode=0 Jul 09 14:45:35.425299 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.425192 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" Jul 09 14:45:35.425299 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.425223 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" event={"ID":"813c361b-e9be-4a20-bb81-7ca890c21e6c","Type":"ContainerDied","Data":"32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d"} Jul 09 14:45:35.425299 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.425277 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp" event={"ID":"813c361b-e9be-4a20-bb81-7ca890c21e6c","Type":"ContainerDied","Data":"9203037ab228474eb6ec99a10598211f799b911e37703ba08ad5a31f2f353b90"} Jul 09 14:45:35.425403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.425301 2556 scope.go:117] "RemoveContainer" containerID="32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d" Jul 09 14:45:35.434160 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.434144 2556 scope.go:117] "RemoveContainer" containerID="32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d" Jul 09 14:45:35.434395 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:45:35.434373 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d\": container with ID starting with 32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d not found: ID does not exist" containerID="32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d" Jul 09 14:45:35.434492 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.434403 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d"} err="failed to get container status \"32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d\": rpc error: code = NotFound desc = could not find container \"32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d\": container with ID starting with 32cd683b31582f13f5e0a359252e32c86851936d1de048c50f74afbc2812d66d not found: ID does not exist" Jul 09 14:45:35.446378 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.446355 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp"] Jul 09 14:45:35.452949 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.452916 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-577c63-74769d558b-k9hvp"] Jul 09 14:45:35.463507 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.463490 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65jbw\" (UniqueName: \"kubernetes.io/projected/813c361b-e9be-4a20-bb81-7ca890c21e6c-kube-api-access-65jbw\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:45:35.463575 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.463510 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813c361b-e9be-4a20-bb81-7ca890c21e6c-openshift-service-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:45:35.463575 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:35.463521 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813c361b-e9be-4a20-bb81-7ca890c21e6c-maas-api-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:45:36.075182 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:36.075151 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="813c361b-e9be-4a20-bb81-7ca890c21e6c" path="/var/lib/kubelet/pods/813c361b-e9be-4a20-bb81-7ca890c21e6c/volumes" Jul 09 14:45:41.059643 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:41.059608 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:41.132199 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:41.132169 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:41.338208 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:41.338137 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:41.603153 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:41.603068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:41.763047 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:41.763006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:42.200841 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:42.200795 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:42.301909 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:42.301609 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:42.479379 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:42.479282 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:48.157062 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:48.157032 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:48.258514 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:48.258487 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:48.451047 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:48.450950 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:48.697444 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:48.697392 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:49.070624 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:49.070596 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:49.401861 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:49.401761 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:49.580032 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:49.580001 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:52.877882 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:52.877846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:52.968025 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:52.967982 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:53.077252 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.077217 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk"] Jul 09 14:45:53.077778 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.077762 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813c361b-e9be-4a20-bb81-7ca890c21e6c" containerName="maas-api" Jul 09 14:45:53.077838 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.077781 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="813c361b-e9be-4a20-bb81-7ca890c21e6c" containerName="maas-api" Jul 09 14:45:53.077879 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.077862 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="813c361b-e9be-4a20-bb81-7ca890c21e6c" containerName="maas-api" Jul 09 14:45:53.080739 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.080722 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.083303 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.083267 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-b22ad3-serving-cert\"" Jul 09 14:45:53.088779 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.088757 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk"] Jul 09 14:45:53.110926 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.110901 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:53.219909 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.219842 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6865d5f0-c6f8-4658-a873-5b8ec2365223-maas-api-tls\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.219909 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.219879 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6865d5f0-c6f8-4658-a873-5b8ec2365223-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.220111 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.219940 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdrf\" (UniqueName: \"kubernetes.io/projected/6865d5f0-c6f8-4658-a873-5b8ec2365223-kube-api-access-xjdrf\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.241631 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.241611 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:53.321209 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.321177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6865d5f0-c6f8-4658-a873-5b8ec2365223-maas-api-tls\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.321366 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.321218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6865d5f0-c6f8-4658-a873-5b8ec2365223-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.321366 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.321287 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdrf\" (UniqueName: \"kubernetes.io/projected/6865d5f0-c6f8-4658-a873-5b8ec2365223-kube-api-access-xjdrf\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.321912 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.321887 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6865d5f0-c6f8-4658-a873-5b8ec2365223-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.323643 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.323625 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6865d5f0-c6f8-4658-a873-5b8ec2365223-maas-api-tls\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.329694 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.329673 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdrf\" (UniqueName: \"kubernetes.io/projected/6865d5f0-c6f8-4658-a873-5b8ec2365223-kube-api-access-xjdrf\") pod \"maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.392601 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.392572 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:53.518025 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.517998 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:53.532204 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.532088 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk"] Jul 09 14:45:53.534625 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:45:53.534596 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6865d5f0_c6f8_4658_a873_5b8ec2365223.slice/crio-77ad057e3c1962bc68b4f2d93ecb7f80342219d90a2d3a9fffdeb7c1ae0327ff WatchSource:0}: Error finding container 77ad057e3c1962bc68b4f2d93ecb7f80342219d90a2d3a9fffdeb7c1ae0327ff: Status 404 returned error can't find the container with id 77ad057e3c1962bc68b4f2d93ecb7f80342219d90a2d3a9fffdeb7c1ae0327ff Jul 09 14:45:53.618045 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:53.618021 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:54.516780 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:54.516742 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" event={"ID":"6865d5f0-c6f8-4658-a873-5b8ec2365223","Type":"ContainerStarted","Data":"1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6"} Jul 09 14:45:54.517177 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:54.516800 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" event={"ID":"6865d5f0-c6f8-4658-a873-5b8ec2365223","Type":"ContainerStarted","Data":"77ad057e3c1962bc68b4f2d93ecb7f80342219d90a2d3a9fffdeb7c1ae0327ff"} Jul 09 14:45:54.517177 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:54.516861 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:45:54.533535 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:54.533480 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" podStartSLOduration=1.09479119 podStartE2EDuration="1.533463846s" totalImagesPullingTime="438.672656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:54.531984215 +0000 UTC m=+1875.136498917" watchObservedRunningTime="2026-07-09 14:45:54.533463846 +0000 UTC m=+1875.137978546" Jul 09 14:45:57.438664 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:57.438626 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:57.800089 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:57.800059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:58.110048 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:58.109956 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:45:58.264068 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:45:58.264026 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:00.527186 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:00.527158 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:46:00.814090 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:00.814059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:00.893128 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:00.893098 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:00.960321 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:00.960295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:01.192524 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:01.192443 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:01.367521 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:01.367446 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:01.465523 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:01.465441 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:03.709981 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:03.709945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:05.371904 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:05.371853 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:15.183359 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:15.183325 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:15.289191 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:15.289161 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:17.307503 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:17.307468 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:17.493531 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:17.493495 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:19.215057 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:19.215010 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:22.207462 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:22.207404 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:22.304512 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:22.304482 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:37.499709 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.499671 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx"] Jul 09 14:46:37.503224 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.503203 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.505892 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.505871 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"default-dockercfg-kqnqp\"" Jul 09 14:46:37.506241 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.506222 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"openshift-service-ca.crt\"" Jul 09 14:46:37.506692 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.506674 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"kube-root-ca.crt\"" Jul 09 14:46:37.506825 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.506806 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"auth-test-model-c4eda7-kserve-self-signed-certs\"" Jul 09 14:46:37.523063 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.523041 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx"] Jul 09 14:46:37.600455 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.600410 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-home\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.600597 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.600463 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-model-cache\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.600597 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.600551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-dshm\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.600701 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.600598 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b1088fb-3785-4698-ab86-6f090fc2693b-tls-certs\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.600701 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.600661 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wc8\" (UniqueName: \"kubernetes.io/projected/3b1088fb-3785-4698-ab86-6f090fc2693b-kube-api-access-h8wc8\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.600779 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.600715 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-tmp-dir\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701300 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701263 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wc8\" (UniqueName: \"kubernetes.io/projected/3b1088fb-3785-4698-ab86-6f090fc2693b-kube-api-access-h8wc8\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701492 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-tmp-dir\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701492 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701374 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-home\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701492 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701404 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-model-cache\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701492 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-dshm\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701729 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701519 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b1088fb-3785-4698-ab86-6f090fc2693b-tls-certs\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701834 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701811 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-home\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701834 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701827 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-tmp-dir\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.701940 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.701896 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-model-cache\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.703625 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.703604 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-dshm\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.704044 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.704023 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b1088fb-3785-4698-ab86-6f090fc2693b-tls-certs\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.714728 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.714703 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wc8\" (UniqueName: \"kubernetes.io/projected/3b1088fb-3785-4698-ab86-6f090fc2693b-kube-api-access-h8wc8\") pod \"auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.720232 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.720204 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:37.812903 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.812867 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:37.959263 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:37.959239 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx"] Jul 09 14:46:37.961254 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:46:37.961225 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b1088fb_3785_4698_ab86_6f090fc2693b.slice/crio-526077fca1812cf323e04fc8a34aad45df64651f5d91a3d7778e180a0bd36450 WatchSource:0}: Error finding container 526077fca1812cf323e04fc8a34aad45df64651f5d91a3d7778e180a0bd36450: Status 404 returned error can't find the container with id 526077fca1812cf323e04fc8a34aad45df64651f5d91a3d7778e180a0bd36450 Jul 09 14:46:38.722698 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:38.722664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" event={"ID":"3b1088fb-3785-4698-ab86-6f090fc2693b","Type":"ContainerStarted","Data":"2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331"} Jul 09 14:46:38.723064 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:38.722708 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" event={"ID":"3b1088fb-3785-4698-ab86-6f090fc2693b","Type":"ContainerStarted","Data":"526077fca1812cf323e04fc8a34aad45df64651f5d91a3d7778e180a0bd36450"} Jul 09 14:46:38.748347 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:38.748298 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" podStartSLOduration=1.748283143 podStartE2EDuration="1.748283143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:46:38.746374577 +0000 UTC m=+1919.350889276" watchObservedRunningTime="2026-07-09 14:46:38.748283143 +0000 UTC m=+1919.352797842" Jul 09 14:46:47.813526 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:47.813487 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:47.814106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:47.813541 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:47.825872 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:47.825848 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:48.778771 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:48.778745 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:46:54.461129 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:54.461100 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:46:54.525350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:46:54.525319 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:47:36.438346 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:47:36.438266 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:48:25.632230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:48:25.632198 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:48:44.741244 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:48:44.741210 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:48:45.549646 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:48:45.549616 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx"] Jul 09 14:48:45.576250 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:48:45.576224 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:48:50.314909 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:48:50.314872 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:49:00.569350 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.569313 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" podUID="3b1088fb-3785-4698-ab86-6f090fc2693b" containerName="main" containerID="cri-o://2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331" gracePeriod=45 Jul 09 14:49:00.822998 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.822941 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:49:00.870893 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.870861 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-tmp-dir\") pod \"3b1088fb-3785-4698-ab86-6f090fc2693b\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " Jul 09 14:49:00.871075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.870911 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-dshm\") pod \"3b1088fb-3785-4698-ab86-6f090fc2693b\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " Jul 09 14:49:00.871075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.870931 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8wc8\" (UniqueName: \"kubernetes.io/projected/3b1088fb-3785-4698-ab86-6f090fc2693b-kube-api-access-h8wc8\") pod \"3b1088fb-3785-4698-ab86-6f090fc2693b\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " Jul 09 14:49:00.871075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.870968 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-model-cache\") pod \"3b1088fb-3785-4698-ab86-6f090fc2693b\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " Jul 09 14:49:00.871075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.870997 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-home\") pod \"3b1088fb-3785-4698-ab86-6f090fc2693b\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " Jul 09 14:49:00.871075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.871020 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b1088fb-3785-4698-ab86-6f090fc2693b-tls-certs\") pod \"3b1088fb-3785-4698-ab86-6f090fc2693b\" (UID: \"3b1088fb-3785-4698-ab86-6f090fc2693b\") " Jul 09 14:49:00.871403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.871165 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b1088fb-3785-4698-ab86-6f090fc2693b" (UID: "3b1088fb-3785-4698-ab86-6f090fc2693b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:00.871403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.871292 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-home" (OuterVolumeSpecName: "home") pod "3b1088fb-3785-4698-ab86-6f090fc2693b" (UID: "3b1088fb-3785-4698-ab86-6f090fc2693b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:00.871403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.871302 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b1088fb-3785-4698-ab86-6f090fc2693b" (UID: "3b1088fb-3785-4698-ab86-6f090fc2693b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:00.871403 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.871339 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-tmp-dir\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:49:00.873398 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.873373 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-dshm" (OuterVolumeSpecName: "dshm") pod "3b1088fb-3785-4698-ab86-6f090fc2693b" (UID: "3b1088fb-3785-4698-ab86-6f090fc2693b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:00.873398 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.873382 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1088fb-3785-4698-ab86-6f090fc2693b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b1088fb-3785-4698-ab86-6f090fc2693b" (UID: "3b1088fb-3785-4698-ab86-6f090fc2693b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:49:00.873553 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.873492 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1088fb-3785-4698-ab86-6f090fc2693b-kube-api-access-h8wc8" (OuterVolumeSpecName: "kube-api-access-h8wc8") pod "3b1088fb-3785-4698-ab86-6f090fc2693b" (UID: "3b1088fb-3785-4698-ab86-6f090fc2693b"). InnerVolumeSpecName "kube-api-access-h8wc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:49:00.972514 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.972469 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-dshm\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:49:00.972514 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.972508 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wc8\" (UniqueName: \"kubernetes.io/projected/3b1088fb-3785-4698-ab86-6f090fc2693b-kube-api-access-h8wc8\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:49:00.972514 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.972520 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-model-cache\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:49:00.972514 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.972529 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b1088fb-3785-4698-ab86-6f090fc2693b-home\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:49:00.972769 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:00.972537 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b1088fb-3785-4698-ab86-6f090fc2693b-tls-certs\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:49:01.378252 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.378168 2556 generic.go:356] "Generic (PLEG): container finished" podID="3b1088fb-3785-4698-ab86-6f090fc2693b" containerID="2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331" exitCode=0 Jul 09 14:49:01.378252 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.378212 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" Jul 09 14:49:01.378515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.378251 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" event={"ID":"3b1088fb-3785-4698-ab86-6f090fc2693b","Type":"ContainerDied","Data":"2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331"} Jul 09 14:49:01.378515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.378296 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx" event={"ID":"3b1088fb-3785-4698-ab86-6f090fc2693b","Type":"ContainerDied","Data":"526077fca1812cf323e04fc8a34aad45df64651f5d91a3d7778e180a0bd36450"} Jul 09 14:49:01.378515 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.378314 2556 scope.go:117] "RemoveContainer" containerID="2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331" Jul 09 14:49:01.387107 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.387091 2556 scope.go:117] "RemoveContainer" containerID="2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331" Jul 09 14:49:01.387361 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:49:01.387343 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331\": container with ID starting with 2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331 not found: ID does not exist" containerID="2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331" Jul 09 14:49:01.387442 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.387368 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331"} err="failed to get container status \"2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331\": rpc error: code = NotFound desc = could not find container \"2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331\": container with ID starting with 2656202bc9659eefd3575c3e8868eb2867b6c01229bfd16b50553ba396431331 not found: ID does not exist" Jul 09 14:49:01.397932 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.397910 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx"] Jul 09 14:49:01.401845 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:01.401826 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/auth-test-model-c4eda7-kserve-c69b97cd7-mjnfx"] Jul 09 14:49:02.076099 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:02.076061 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1088fb-3785-4698-ab86-6f090fc2693b" path="/var/lib/kubelet/pods/3b1088fb-3785-4698-ab86-6f090fc2693b/volumes" Jul 09 14:49:27.472304 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.472264 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s"] Jul 09 14:49:27.472792 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.472651 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1088fb-3785-4698-ab86-6f090fc2693b" containerName="main" Jul 09 14:49:27.472792 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.472664 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1088fb-3785-4698-ab86-6f090fc2693b" containerName="main" Jul 09 14:49:27.472792 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.472719 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1088fb-3785-4698-ab86-6f090fc2693b" containerName="main" Jul 09 14:49:27.475914 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.475893 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.479091 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.479046 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"kube-root-ca.crt\"" Jul 09 14:49:27.479091 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.479071 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"openshift-service-ca.crt\"" Jul 09 14:49:27.479270 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.479135 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"sub-test-model-c4eda7-kserve-self-signed-certs\"" Jul 09 14:49:27.479270 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.479143 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c4eda7\"/\"default-dockercfg-kqnqp\"" Jul 09 14:49:27.484270 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.484248 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s"] Jul 09 14:49:27.503007 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.502975 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:49:27.612217 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.612173 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14225c0c-6fb3-4759-b862-f4983abf3767-tls-certs\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.612386 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.612233 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxk5m\" (UniqueName: \"kubernetes.io/projected/14225c0c-6fb3-4759-b862-f4983abf3767-kube-api-access-qxk5m\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.612386 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.612289 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-model-cache\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.612386 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.612344 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-tmp-dir\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.612547 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.612449 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-home\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.612547 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.612493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-dshm\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.713561 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.713520 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14225c0c-6fb3-4759-b862-f4983abf3767-tls-certs\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.713561 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.713563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxk5m\" (UniqueName: \"kubernetes.io/projected/14225c0c-6fb3-4759-b862-f4983abf3767-kube-api-access-qxk5m\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.713817 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.713613 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-model-cache\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.713817 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.713654 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-tmp-dir\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.713817 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.713710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-home\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.713817 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.713748 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-dshm\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.714205 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.714179 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-home\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.714305 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.714247 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-model-cache\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.714369 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.714321 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-tmp-dir\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.716117 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.716097 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14225c0c-6fb3-4759-b862-f4983abf3767-tls-certs\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.716181 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.716138 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-dshm\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.721902 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.721881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxk5m\" (UniqueName: \"kubernetes.io/projected/14225c0c-6fb3-4759-b862-f4983abf3767-kube-api-access-qxk5m\") pod \"sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.786088 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.786011 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:27.921674 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:27.921564 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s"] Jul 09 14:49:27.924533 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:49:27.924490 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14225c0c_6fb3_4759_b862_f4983abf3767.slice/crio-77cd7b427b58a5cdf4ae294707656afe951a5e57573825925eb09ea00eeb47de WatchSource:0}: Error finding container 77cd7b427b58a5cdf4ae294707656afe951a5e57573825925eb09ea00eeb47de: Status 404 returned error can't find the container with id 77cd7b427b58a5cdf4ae294707656afe951a5e57573825925eb09ea00eeb47de Jul 09 14:49:28.505042 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:28.505010 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" event={"ID":"14225c0c-6fb3-4759-b862-f4983abf3767","Type":"ContainerStarted","Data":"d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9"} Jul 09 14:49:28.505042 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:28.505058 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" event={"ID":"14225c0c-6fb3-4759-b862-f4983abf3767","Type":"ContainerStarted","Data":"77cd7b427b58a5cdf4ae294707656afe951a5e57573825925eb09ea00eeb47de"} Jul 09 14:49:28.528200 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:28.528152 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" podStartSLOduration=1.528138197 podStartE2EDuration="1.528138197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:28.52770594 +0000 UTC m=+2089.132220640" watchObservedRunningTime="2026-07-09 14:49:28.528138197 +0000 UTC m=+2089.132652920" Jul 09 14:49:37.786413 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:37.786367 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:37.786928 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:37.786444 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:37.798997 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:37.798974 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:38.566857 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:38.566819 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:49:40.021987 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:40.021953 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:49:40.028035 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:49:40.028015 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hhjnx_e5ccd446-9cc6-4a7f-b9ae-4fba80f562fe/ovn-acl-logging/0.log" Jul 09 14:50:20.372956 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:20.372923 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:50:21.319883 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:21.319850 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s"] Jul 09 14:50:21.349596 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:21.349559 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:50:25.906544 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.906514 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj"] Jul 09 14:50:25.910008 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.909990 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:25.913323 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.913299 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b22ad3\"/\"rate-test-model-b22ad3-kserve-self-signed-certs\"" Jul 09 14:50:25.914316 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.914292 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b22ad3\"/\"default-dockercfg-ltx96\"" Jul 09 14:50:25.926153 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.926135 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b22ad3\"/\"openshift-service-ca.crt\"" Jul 09 14:50:25.926240 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.926153 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b22ad3\"/\"kube-root-ca.crt\"" Jul 09 14:50:25.927166 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.927144 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:50:25.942873 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:25.942852 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj"] Jul 09 14:50:26.024318 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.024290 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-model-cache\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.024477 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.024324 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pw9j\" (UniqueName: \"kubernetes.io/projected/fe26241c-1fbe-48be-bc98-76bfac7ad551-kube-api-access-7pw9j\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.024477 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.024349 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-dshm\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.024477 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.024462 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-tmp-dir\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.024599 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.024517 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-home\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.024599 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.024582 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe26241c-1fbe-48be-bc98-76bfac7ad551-tls-certs\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.125925 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.125886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-tmp-dir\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.125948 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-home\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.125984 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe26241c-1fbe-48be-bc98-76bfac7ad551-tls-certs\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.126027 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-model-cache\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126122 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.126048 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pw9j\" (UniqueName: \"kubernetes.io/projected/fe26241c-1fbe-48be-bc98-76bfac7ad551-kube-api-access-7pw9j\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126286 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.126132 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-dshm\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126374 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.126351 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-tmp-dir\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126415 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.126379 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-home\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.126487 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.126454 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-model-cache\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.128231 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.128203 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-dshm\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.128578 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.128561 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe26241c-1fbe-48be-bc98-76bfac7ad551-tls-certs\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.136139 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.136117 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pw9j\" (UniqueName: \"kubernetes.io/projected/fe26241c-1fbe-48be-bc98-76bfac7ad551-kube-api-access-7pw9j\") pod \"rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.220037 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.219967 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:26.353680 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.353636 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj"] Jul 09 14:50:26.356250 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:50:26.356222 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe26241c_1fbe_48be_bc98_76bfac7ad551.slice/crio-2fa751481e6b48f564e83aba80c2148f599a9f07c3850923b6ad0b7ae381029c WatchSource:0}: Error finding container 2fa751481e6b48f564e83aba80c2148f599a9f07c3850923b6ad0b7ae381029c: Status 404 returned error can't find the container with id 2fa751481e6b48f564e83aba80c2148f599a9f07c3850923b6ad0b7ae381029c Jul 09 14:50:26.780078 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.780045 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" event={"ID":"fe26241c-1fbe-48be-bc98-76bfac7ad551","Type":"ContainerStarted","Data":"825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92"} Jul 09 14:50:26.780078 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.780084 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" event={"ID":"fe26241c-1fbe-48be-bc98-76bfac7ad551","Type":"ContainerStarted","Data":"2fa751481e6b48f564e83aba80c2148f599a9f07c3850923b6ad0b7ae381029c"} Jul 09 14:50:26.804231 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:26.804184 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" podStartSLOduration=1.804167326 podStartE2EDuration="1.804167326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:26.80270834 +0000 UTC m=+2147.407223041" watchObservedRunningTime="2026-07-09 14:50:26.804167326 +0000 UTC m=+2147.408682026" Jul 09 14:50:36.220376 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.220285 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:36.220376 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.220338 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:36.232950 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.232926 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:36.339398 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.339343 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" podUID="14225c0c-6fb3-4759-b862-f4983abf3767" containerName="main" containerID="cri-o://d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9" gracePeriod=45 Jul 09 14:50:36.589786 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.589764 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:50:36.603931 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.603906 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14225c0c-6fb3-4759-b862-f4983abf3767-tls-certs\") pod \"14225c0c-6fb3-4759-b862-f4983abf3767\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " Jul 09 14:50:36.604076 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.603943 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxk5m\" (UniqueName: \"kubernetes.io/projected/14225c0c-6fb3-4759-b862-f4983abf3767-kube-api-access-qxk5m\") pod \"14225c0c-6fb3-4759-b862-f4983abf3767\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " Jul 09 14:50:36.604076 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.603988 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-tmp-dir\") pod \"14225c0c-6fb3-4759-b862-f4983abf3767\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " Jul 09 14:50:36.604265 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604200 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-model-cache\") pod \"14225c0c-6fb3-4759-b862-f4983abf3767\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " Jul 09 14:50:36.604265 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604245 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-home\") pod \"14225c0c-6fb3-4759-b862-f4983abf3767\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " Jul 09 14:50:36.604445 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604281 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14225c0c-6fb3-4759-b862-f4983abf3767" (UID: "14225c0c-6fb3-4759-b862-f4983abf3767"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:50:36.604445 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604286 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-dshm\") pod \"14225c0c-6fb3-4759-b862-f4983abf3767\" (UID: \"14225c0c-6fb3-4759-b862-f4983abf3767\") " Jul 09 14:50:36.604574 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604455 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-model-cache" (OuterVolumeSpecName: "model-cache") pod "14225c0c-6fb3-4759-b862-f4983abf3767" (UID: "14225c0c-6fb3-4759-b862-f4983abf3767"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:50:36.604574 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604552 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-home" (OuterVolumeSpecName: "home") pod "14225c0c-6fb3-4759-b862-f4983abf3767" (UID: "14225c0c-6fb3-4759-b862-f4983abf3767"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:50:36.604697 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604686 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-model-cache\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:50:36.604753 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604704 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-home\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:50:36.604753 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.604718 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-tmp-dir\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:50:36.606132 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.606107 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14225c0c-6fb3-4759-b862-f4983abf3767-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14225c0c-6fb3-4759-b862-f4983abf3767" (UID: "14225c0c-6fb3-4759-b862-f4983abf3767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:50:36.606232 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.606152 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14225c0c-6fb3-4759-b862-f4983abf3767-kube-api-access-qxk5m" (OuterVolumeSpecName: "kube-api-access-qxk5m") pod "14225c0c-6fb3-4759-b862-f4983abf3767" (UID: "14225c0c-6fb3-4759-b862-f4983abf3767"). InnerVolumeSpecName "kube-api-access-qxk5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:50:36.606232 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.606153 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-dshm" (OuterVolumeSpecName: "dshm") pod "14225c0c-6fb3-4759-b862-f4983abf3767" (UID: "14225c0c-6fb3-4759-b862-f4983abf3767"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:50:36.705616 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.705594 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14225c0c-6fb3-4759-b862-f4983abf3767-dshm\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:50:36.705616 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.705614 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14225c0c-6fb3-4759-b862-f4983abf3767-tls-certs\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:50:36.705789 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.705624 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxk5m\" (UniqueName: \"kubernetes.io/projected/14225c0c-6fb3-4759-b862-f4983abf3767-kube-api-access-qxk5m\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:50:36.828576 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.828541 2556 generic.go:356] "Generic (PLEG): container finished" podID="14225c0c-6fb3-4759-b862-f4983abf3767" containerID="d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9" exitCode=0 Jul 09 14:50:36.828743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.828588 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" Jul 09 14:50:36.828743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.828627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" event={"ID":"14225c0c-6fb3-4759-b862-f4983abf3767","Type":"ContainerDied","Data":"d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9"} Jul 09 14:50:36.828743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.828672 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s" event={"ID":"14225c0c-6fb3-4759-b862-f4983abf3767","Type":"ContainerDied","Data":"77cd7b427b58a5cdf4ae294707656afe951a5e57573825925eb09ea00eeb47de"} Jul 09 14:50:36.828743 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.828691 2556 scope.go:117] "RemoveContainer" containerID="d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9" Jul 09 14:50:36.837142 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.837121 2556 scope.go:117] "RemoveContainer" containerID="d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9" Jul 09 14:50:36.837415 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:50:36.837393 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9\": container with ID starting with d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9 not found: ID does not exist" containerID="d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9" Jul 09 14:50:36.837506 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.837439 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9"} err="failed to get container status \"d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9\": rpc error: code = NotFound desc = could not find container \"d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9\": container with ID starting with d0f610b3ecf9793aac199f619ef33b675df37d4427fc2413f59d5c0750df00a9 not found: ID does not exist" Jul 09 14:50:36.843159 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.843136 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:50:36.852982 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.852948 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s"] Jul 09 14:50:36.855253 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:36.855231 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c4eda7/sub-test-model-c4eda7-kserve-6db45c95d7-4mn2s"] Jul 09 14:50:38.075845 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:50:38.075812 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14225c0c-6fb3-4759-b862-f4983abf3767" path="/var/lib/kubelet/pods/14225c0c-6fb3-4759-b862-f4983abf3767/volumes" Jul 09 14:51:02.474792 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:02.474756 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:15.937176 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:15.937148 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:16.036851 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:16.036821 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:18.321231 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:18.321200 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:18.420082 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:18.420051 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:23.053345 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:23.053309 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:23.151828 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:23.151796 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:52.851522 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:52.851489 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj"] Jul 09 14:51:52.872801 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:52.872773 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:51:53.891079 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:51:53.891042 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:07.871813 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:07.871750 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" podUID="fe26241c-1fbe-48be-bc98-76bfac7ad551" containerName="main" containerID="cri-o://825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92" gracePeriod=45 Jul 09 14:52:08.122076 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.122019 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:52:08.217662 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.217632 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-tmp-dir\") pod \"fe26241c-1fbe-48be-bc98-76bfac7ad551\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " Jul 09 14:52:08.217844 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.217670 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-home\") pod \"fe26241c-1fbe-48be-bc98-76bfac7ad551\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " Jul 09 14:52:08.217844 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.217723 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pw9j\" (UniqueName: \"kubernetes.io/projected/fe26241c-1fbe-48be-bc98-76bfac7ad551-kube-api-access-7pw9j\") pod \"fe26241c-1fbe-48be-bc98-76bfac7ad551\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " Jul 09 14:52:08.217844 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.217749 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-dshm\") pod \"fe26241c-1fbe-48be-bc98-76bfac7ad551\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " Jul 09 14:52:08.217844 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.217792 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-model-cache\") pod \"fe26241c-1fbe-48be-bc98-76bfac7ad551\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " Jul 09 14:52:08.218064 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.217917 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe26241c-1fbe-48be-bc98-76bfac7ad551" (UID: "fe26241c-1fbe-48be-bc98-76bfac7ad551"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:08.218064 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.217932 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe26241c-1fbe-48be-bc98-76bfac7ad551-tls-certs\") pod \"fe26241c-1fbe-48be-bc98-76bfac7ad551\" (UID: \"fe26241c-1fbe-48be-bc98-76bfac7ad551\") " Jul 09 14:52:08.218064 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.218015 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe26241c-1fbe-48be-bc98-76bfac7ad551" (UID: "fe26241c-1fbe-48be-bc98-76bfac7ad551"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:08.218064 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.218033 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-home" (OuterVolumeSpecName: "home") pod "fe26241c-1fbe-48be-bc98-76bfac7ad551" (UID: "fe26241c-1fbe-48be-bc98-76bfac7ad551"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:08.218273 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.218249 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-model-cache\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:08.218324 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.218277 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-tmp-dir\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:08.218324 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.218290 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-home\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:08.220067 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.220040 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe26241c-1fbe-48be-bc98-76bfac7ad551-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe26241c-1fbe-48be-bc98-76bfac7ad551" (UID: "fe26241c-1fbe-48be-bc98-76bfac7ad551"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:52:08.220183 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.220078 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-dshm" (OuterVolumeSpecName: "dshm") pod "fe26241c-1fbe-48be-bc98-76bfac7ad551" (UID: "fe26241c-1fbe-48be-bc98-76bfac7ad551"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:52:08.220566 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.220540 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe26241c-1fbe-48be-bc98-76bfac7ad551-kube-api-access-7pw9j" (OuterVolumeSpecName: "kube-api-access-7pw9j") pod "fe26241c-1fbe-48be-bc98-76bfac7ad551" (UID: "fe26241c-1fbe-48be-bc98-76bfac7ad551"). InnerVolumeSpecName "kube-api-access-7pw9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:08.252619 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.252594 2556 generic.go:356] "Generic (PLEG): container finished" podID="fe26241c-1fbe-48be-bc98-76bfac7ad551" containerID="825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92" exitCode=0 Jul 09 14:52:08.252716 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.252635 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" Jul 09 14:52:08.252716 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.252679 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" event={"ID":"fe26241c-1fbe-48be-bc98-76bfac7ad551","Type":"ContainerDied","Data":"825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92"} Jul 09 14:52:08.252806 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.252718 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj" event={"ID":"fe26241c-1fbe-48be-bc98-76bfac7ad551","Type":"ContainerDied","Data":"2fa751481e6b48f564e83aba80c2148f599a9f07c3850923b6ad0b7ae381029c"} Jul 09 14:52:08.252806 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.252734 2556 scope.go:117] "RemoveContainer" containerID="825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92" Jul 09 14:52:08.263964 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.263946 2556 scope.go:117] "RemoveContainer" containerID="825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92" Jul 09 14:52:08.264249 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:52:08.264229 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92\": container with ID starting with 825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92 not found: ID does not exist" containerID="825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92" Jul 09 14:52:08.264323 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.264256 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92"} err="failed to get container status \"825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92\": rpc error: code = NotFound desc = could not find container \"825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92\": container with ID starting with 825bc53c3bd669f8907ddb3efa19eb2692b64b513e99d5541b2778d66d940b92 not found: ID does not exist" Jul 09 14:52:08.276653 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.276632 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj"] Jul 09 14:52:08.281315 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.281293 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b22ad3/rate-test-model-b22ad3-kserve-56fd9d9cc5-bplvj"] Jul 09 14:52:08.319660 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.319636 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe26241c-1fbe-48be-bc98-76bfac7ad551-tls-certs\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:08.319660 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.319657 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pw9j\" (UniqueName: \"kubernetes.io/projected/fe26241c-1fbe-48be-bc98-76bfac7ad551-kube-api-access-7pw9j\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:08.319784 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:08.319669 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe26241c-1fbe-48be-bc98-76bfac7ad551-dshm\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:10.075905 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:10.075867 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe26241c-1fbe-48be-bc98-76bfac7ad551" path="/var/lib/kubelet/pods/fe26241c-1fbe-48be-bc98-76bfac7ad551/volumes" Jul 09 14:52:11.576191 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:11.576158 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:13.475200 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:13.475166 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:13.888012 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:13.887974 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:14.056685 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.056648 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:14.160959 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.160856 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:14.328791 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.328759 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk"] Jul 09 14:52:14.328993 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.328971 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" podUID="6865d5f0-c6f8-4658-a873-5b8ec2365223" containerName="maas-api" containerID="cri-o://1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6" gracePeriod=30 Jul 09 14:52:14.390565 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.390536 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:14.575244 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.575220 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:52:14.670855 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.670824 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjdrf\" (UniqueName: \"kubernetes.io/projected/6865d5f0-c6f8-4658-a873-5b8ec2365223-kube-api-access-xjdrf\") pod \"6865d5f0-c6f8-4658-a873-5b8ec2365223\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " Jul 09 14:52:14.671014 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.670861 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6865d5f0-c6f8-4658-a873-5b8ec2365223-maas-api-tls\") pod \"6865d5f0-c6f8-4658-a873-5b8ec2365223\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " Jul 09 14:52:14.671014 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.670943 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6865d5f0-c6f8-4658-a873-5b8ec2365223-openshift-service-ca\") pod \"6865d5f0-c6f8-4658-a873-5b8ec2365223\" (UID: \"6865d5f0-c6f8-4658-a873-5b8ec2365223\") " Jul 09 14:52:14.671253 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.671226 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6865d5f0-c6f8-4658-a873-5b8ec2365223-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6865d5f0-c6f8-4658-a873-5b8ec2365223" (UID: "6865d5f0-c6f8-4658-a873-5b8ec2365223"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:52:14.672885 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.672860 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6865d5f0-c6f8-4658-a873-5b8ec2365223-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6865d5f0-c6f8-4658-a873-5b8ec2365223" (UID: "6865d5f0-c6f8-4658-a873-5b8ec2365223"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:52:14.673032 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.673008 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6865d5f0-c6f8-4658-a873-5b8ec2365223-kube-api-access-xjdrf" (OuterVolumeSpecName: "kube-api-access-xjdrf") pod "6865d5f0-c6f8-4658-a873-5b8ec2365223" (UID: "6865d5f0-c6f8-4658-a873-5b8ec2365223"). InnerVolumeSpecName "kube-api-access-xjdrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:52:14.771667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.771603 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6865d5f0-c6f8-4658-a873-5b8ec2365223-openshift-service-ca\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:14.771667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.771624 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdrf\" (UniqueName: \"kubernetes.io/projected/6865d5f0-c6f8-4658-a873-5b8ec2365223-kube-api-access-xjdrf\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:14.771667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:14.771634 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6865d5f0-c6f8-4658-a873-5b8ec2365223-maas-api-tls\") on node \"ip-10-0-142-0.ec2.internal\" DevicePath \"\"" Jul 09 14:52:15.286086 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.286053 2556 generic.go:356] "Generic (PLEG): container finished" podID="6865d5f0-c6f8-4658-a873-5b8ec2365223" containerID="1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6" exitCode=0 Jul 09 14:52:15.286261 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.286101 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" Jul 09 14:52:15.286261 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.286144 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" event={"ID":"6865d5f0-c6f8-4658-a873-5b8ec2365223","Type":"ContainerDied","Data":"1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6"} Jul 09 14:52:15.286261 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.286199 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk" event={"ID":"6865d5f0-c6f8-4658-a873-5b8ec2365223","Type":"ContainerDied","Data":"77ad057e3c1962bc68b4f2d93ecb7f80342219d90a2d3a9fffdeb7c1ae0327ff"} Jul 09 14:52:15.286261 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.286221 2556 scope.go:117] "RemoveContainer" containerID="1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6" Jul 09 14:52:15.294816 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.294799 2556 scope.go:117] "RemoveContainer" containerID="1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6" Jul 09 14:52:15.295052 ip-10-0-142-0 kubenswrapper[2556]: E0709 14:52:15.295033 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6\": container with ID starting with 1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6 not found: ID does not exist" containerID="1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6" Jul 09 14:52:15.295098 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.295060 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6"} err="failed to get container status \"1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6\": rpc error: code = NotFound desc = could not find container \"1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6\": container with ID starting with 1a373d2661df212fad848d6dc4f1a8f72806cca822c14261e76f09ae545681c6 not found: ID does not exist" Jul 09 14:52:15.311353 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.311331 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk"] Jul 09 14:52:15.323567 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:15.323545 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b22ad3-654c6bdf96-pcbsk"] Jul 09 14:52:16.076255 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:16.076220 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6865d5f0-c6f8-4658-a873-5b8ec2365223" path="/var/lib/kubelet/pods/6865d5f0-c6f8-4658-a873-5b8ec2365223/volumes" Jul 09 14:52:20.690442 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:20.690390 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:20.803960 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:20.803935 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:21.065596 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:21.065561 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:21.245095 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:21.245063 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:21.459123 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:21.459028 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:21.806376 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:21.806351 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:21.949588 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:21.949556 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:22.045655 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:22.045624 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:22.261625 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:22.261547 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:28.132020 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:28.131986 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:28.233409 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:28.233375 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:28.462410 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:28.462325 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:28.662510 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:28.662476 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:29.085054 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:29.085022 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b6bbc4f8f-km5k9_7e078cc7-84e6-4f96-9520-663490fad786/authorino/0.log" Jul 09 14:52:34.339762 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:34.339729 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m56jk_3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb/manager/0.log" Jul 09 14:52:34.457191 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:34.457163 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586f64b4d6-w7w5d_b1113875-ac63-4c71-9caa-38c1b7f89a21/manager/0.log" Jul 09 14:52:35.066511 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:35.066482 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-w2fjt_0836e61d-6ac8-4c08-a5af-a5d44d5213fa/copy-manifests/0.log" Jul 09 14:52:35.073608 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:35.073589 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-w2fjt_0836e61d-6ac8-4c08-a5af-a5d44d5213fa/manager/0.log" Jul 09 14:52:35.413858 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:35.413779 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7cfbd6c584-qsmm5_a4dafa70-6e4b-4eaf-b492-3653b2e3b2e7/maas-api/0.log" Jul 09 14:52:36.814211 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:36.814181 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b6bbc4f8f-km5k9_7e078cc7-84e6-4f96-9520-663490fad786/authorino/0.log" Jul 09 14:52:37.185073 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:37.184996 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jg9th_9dc29b52-e93a-45eb-8a26-7f532346f8e8/kuadrant-console-plugin/0.log" Jul 09 14:52:38.263888 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:38.263861 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-28pmh_8d3c671f-941a-4f08-95bf-7102fcc80037/kube-auth-proxy/0.log" Jul 09 14:52:38.629764 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:38.629741 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vjmxz_f8b67823-9b07-449f-a97a-547a3c2ac4c3/payload-pre-processing/0.log" Jul 09 14:52:38.748075 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:38.748046 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ms9cp_b072044b-0a70-4c8b-a3de-a8ecab3cf841/payload-processing/0.log" Jul 09 14:52:39.333106 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:39.333078 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vbz7q_32a556da-20a5-4f2c-b606-125329c25035/main/0.log" Jul 09 14:52:39.572666 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:39.572629 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzd59z_d2720ab2-bba6-4bf5-84d4-55940f8f08f3/main/0.log" Jul 09 14:52:39.689264 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:39.689189 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lxc49_cc49f76b-c7b1-49be-b621-05f71623859e/main/0.log" Jul 09 14:52:42.741346 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:42.741312 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:43.935518 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:43.935480 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:44.032936 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:44.032903 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:52:47.007541 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:47.007502 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tmpjn_e988dcf9-c3bd-4bc1-95d0-d5217096ec97/global-pull-secret-syncer/0.log" Jul 09 14:52:47.077735 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:47.077707 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" Jul 09 14:52:47.206520 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:47.206489 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-0.ec2.internal_23cb67f50f5c5e7503dadf6a1637939a/haproxy/0.log" Jul 09 14:52:52.389929 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:52.389844 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b6bbc4f8f-km5k9_7e078cc7-84e6-4f96-9520-663490fad786/authorino/0.log" Jul 09 14:52:52.502846 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:52.502817 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jg9th_9dc29b52-e93a-45eb-8a26-7f532346f8e8/kuadrant-console-plugin/0.log" Jul 09 14:52:54.778873 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:54.778846 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-f8htr_fda6cf0f-675a-48a5-9b8d-b2e390b37031/kube-state-metrics/0.log" Jul 09 14:52:54.805337 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:54.805316 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-f8htr_fda6cf0f-675a-48a5-9b8d-b2e390b37031/kube-rbac-proxy-main/0.log" Jul 09 14:52:54.832667 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:54.832635 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-f8htr_fda6cf0f-675a-48a5-9b8d-b2e390b37031/kube-rbac-proxy-self/0.log" Jul 09 14:52:54.893669 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:54.893610 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-wp8d2_d0ec1ee5-583c-4438-943f-7dae1d2bc5b5/monitoring-plugin/0.log" Jul 09 14:52:55.103888 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.103861 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pktfk_9b50e9c2-964e-4a44-a882-7476b118808f/node-exporter/0.log" Jul 09 14:52:55.129756 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.129731 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pktfk_9b50e9c2-964e-4a44-a882-7476b118808f/kube-rbac-proxy/0.log" Jul 09 14:52:55.155200 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.155130 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pktfk_9b50e9c2-964e-4a44-a882-7476b118808f/init-textfile/0.log" Jul 09 14:52:55.193880 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.193853 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-sfmh7_591f3a7b-19a2-4ad1-ae4b-39b7e852811b/kube-rbac-proxy-main/0.log" Jul 09 14:52:55.223931 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.223911 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-sfmh7_591f3a7b-19a2-4ad1-ae4b-39b7e852811b/kube-rbac-proxy-self/0.log" Jul 09 14:52:55.247313 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.247295 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-sfmh7_591f3a7b-19a2-4ad1-ae4b-39b7e852811b/openshift-state-metrics/0.log" Jul 09 14:52:55.290867 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.290847 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb62817c-e209-49c6-9b43-b4290add5f80/prometheus/0.log" Jul 09 14:52:55.316089 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.316073 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb62817c-e209-49c6-9b43-b4290add5f80/config-reloader/0.log" Jul 09 14:52:55.340344 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.340328 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb62817c-e209-49c6-9b43-b4290add5f80/thanos-sidecar/0.log" Jul 09 14:52:55.366522 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.366504 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb62817c-e209-49c6-9b43-b4290add5f80/kube-rbac-proxy-web/0.log" Jul 09 14:52:55.387702 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.387683 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb62817c-e209-49c6-9b43-b4290add5f80/kube-rbac-proxy/0.log" Jul 09 14:52:55.410544 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.410494 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb62817c-e209-49c6-9b43-b4290add5f80/kube-rbac-proxy-thanos/0.log" Jul 09 14:52:55.434855 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:55.434838 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eb62817c-e209-49c6-9b43-b4290add5f80/init-config-reloader/0.log" Jul 09 14:52:56.708580 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.708547 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2"] Jul 09 14:52:56.709117 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709097 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6865d5f0-c6f8-4658-a873-5b8ec2365223" containerName="maas-api" Jul 09 14:52:56.709117 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709119 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6865d5f0-c6f8-4658-a873-5b8ec2365223" containerName="maas-api" Jul 09 14:52:56.709254 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709151 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14225c0c-6fb3-4759-b862-f4983abf3767" containerName="main" Jul 09 14:52:56.709254 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709160 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14225c0c-6fb3-4759-b862-f4983abf3767" containerName="main" Jul 09 14:52:56.709254 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709185 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe26241c-1fbe-48be-bc98-76bfac7ad551" containerName="main" Jul 09 14:52:56.709254 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709194 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe26241c-1fbe-48be-bc98-76bfac7ad551" containerName="main" Jul 09 14:52:56.709488 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709274 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe26241c-1fbe-48be-bc98-76bfac7ad551" containerName="main" Jul 09 14:52:56.709488 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709292 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6865d5f0-c6f8-4658-a873-5b8ec2365223" containerName="maas-api" Jul 09 14:52:56.709488 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.709302 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14225c0c-6fb3-4759-b862-f4983abf3767" containerName="main" Jul 09 14:52:56.712669 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.712644 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.715162 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.715144 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mpppr\"/\"openshift-service-ca.crt\"" Jul 09 14:52:56.715268 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.715182 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mpppr\"/\"kube-root-ca.crt\"" Jul 09 14:52:56.716227 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.716193 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mpppr\"/\"default-dockercfg-mbnb2\"" Jul 09 14:52:56.722594 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.722572 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2"] Jul 09 14:52:56.736627 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.736602 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-proc\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.736714 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.736691 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-podres\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.736755 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.736744 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-sys\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.736787 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.736762 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzl2c\" (UniqueName: \"kubernetes.io/projected/244351e0-af3d-4439-8e10-8c7ce80bec64-kube-api-access-zzl2c\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.736876 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.736857 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-lib-modules\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838230 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838200 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-lib-modules\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838365 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-proc\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838365 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-podres\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838365 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-sys\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838361 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzl2c\" (UniqueName: \"kubernetes.io/projected/244351e0-af3d-4439-8e10-8c7ce80bec64-kube-api-access-zzl2c\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838379 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-lib-modules\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838401 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-proc\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838415 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-sys\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.838525 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.838458 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/244351e0-af3d-4439-8e10-8c7ce80bec64-podres\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:56.846817 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:56.846794 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzl2c\" (UniqueName: \"kubernetes.io/projected/244351e0-af3d-4439-8e10-8c7ce80bec64-kube-api-access-zzl2c\") pod \"perf-node-gather-daemonset-v5sw2\" (UID: \"244351e0-af3d-4439-8e10-8c7ce80bec64\") " pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:57.023355 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:57.023286 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:57.149406 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:57.149380 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2"] Jul 09 14:52:57.151739 ip-10-0-142-0 kubenswrapper[2556]: W0709 14:52:57.151711 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod244351e0_af3d_4439_8e10_8c7ce80bec64.slice/crio-b9c90e268a31baba57edd483e3e5c80eb148110d6871e4dda2ceb0215e0c6211 WatchSource:0}: Error finding container b9c90e268a31baba57edd483e3e5c80eb148110d6871e4dda2ceb0215e0c6211: Status 404 returned error can't find the container with id b9c90e268a31baba57edd483e3e5c80eb148110d6871e4dda2ceb0215e0c6211 Jul 09 14:52:57.479983 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:57.479943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" event={"ID":"244351e0-af3d-4439-8e10-8c7ce80bec64","Type":"ContainerStarted","Data":"d1ce7ae26d5df37b0fe4ef398d4d9c63857315298c879c35580f993c44cce567"} Jul 09 14:52:57.480169 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:57.480008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" event={"ID":"244351e0-af3d-4439-8e10-8c7ce80bec64","Type":"ContainerStarted","Data":"b9c90e268a31baba57edd483e3e5c80eb148110d6871e4dda2ceb0215e0c6211"} Jul 09 14:52:57.480169 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:57.480117 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:52:57.499626 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:57.499581 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" podStartSLOduration=1.499568748 podStartE2EDuration="1.499568748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:57.49810157 +0000 UTC m=+2298.102616271" watchObservedRunningTime="2026-07-09 14:52:57.499568748 +0000 UTC m=+2298.104083448" Jul 09 14:52:59.935675 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:59.935650 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2lkd_df2b62f9-61e7-44ed-8880-3772033e8e9c/dns/0.log" Jul 09 14:52:59.960228 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:52:59.960202 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2lkd_df2b62f9-61e7-44ed-8880-3772033e8e9c/kube-rbac-proxy/0.log" Jul 09 14:53:00.069823 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:00.069794 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ch6ct_6a19f14e-5a01-44e1-9933-dca850de7aea/dns-node-resolver/0.log" Jul 09 14:53:00.711073 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:00.711028 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cj6rf_71faa3e7-1beb-4c83-bce1-a3b45a1e8155/node-ca/0.log" Jul 09 14:53:01.852686 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:01.852657 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-28pmh_8d3c671f-941a-4f08-95bf-7102fcc80037/kube-auth-proxy/0.log" Jul 09 14:53:01.976771 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:01.976743 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vjmxz_f8b67823-9b07-449f-a97a-547a3c2ac4c3/payload-pre-processing/0.log" Jul 09 14:53:02.025381 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:02.025353 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ms9cp_b072044b-0a70-4c8b-a3de-a8ecab3cf841/payload-processing/0.log" Jul 09 14:53:02.835605 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:02.835577 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vcf89_c3ea1425-a7d9-4581-b20e-44e0560c8cae/serve-healthcheck-canary/0.log" Jul 09 14:53:03.443583 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:03.443552 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nd8dl_e1aae7da-8e11-42fc-b740-ee4e35e26793/kube-rbac-proxy/0.log" Jul 09 14:53:03.466824 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:03.466800 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nd8dl_e1aae7da-8e11-42fc-b740-ee4e35e26793/exporter/0.log" Jul 09 14:53:03.489641 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:03.489618 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nd8dl_e1aae7da-8e11-42fc-b740-ee4e35e26793/extractor/0.log" Jul 09 14:53:03.494294 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:03.494275 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mpppr/perf-node-gather-daemonset-v5sw2" Jul 09 14:53:04.877488 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:04.877453 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9g5l_911301db-2aff-4ec6-ab8a-f613769de8e0/konnectivity-agent/0.log" limit=1024 Jul 09 14:53:06.103955 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:06.103924 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m56jk_3d52b5de-6d7d-4dbf-bcd5-784be6cd91fb/manager/0.log" Jul 09 14:53:06.201882 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:06.201856 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586f64b4d6-w7w5d_b1113875-ac63-4c71-9caa-38c1b7f89a21/manager/0.log" Jul 09 14:53:06.517664 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:06.517592 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-w2fjt_0836e61d-6ac8-4c08-a5af-a5d44d5213fa/manager/0.log" Jul 09 14:53:06.547216 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:06.547193 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8568cbd5-w2fjt_0836e61d-6ac8-4c08-a5af-a5d44d5213fa/copy-manifests/0.log" Jul 09 14:53:07.559386 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:07.559359 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-28pmh_8d3c671f-941a-4f08-95bf-7102fcc80037/kube-auth-proxy/0.log" Jul 09 14:53:07.660888 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:07.660859 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vjmxz_f8b67823-9b07-449f-a97a-547a3c2ac4c3/payload-pre-processing/0.log" Jul 09 14:53:07.695650 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:07.695616 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ms9cp_b072044b-0a70-4c8b-a3de-a8ecab3cf841/payload-processing/0.log" Jul 09 14:53:07.768873 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:07.768843 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5578c64b55-7j7qn_0be37139-d007-4d00-b46f-60c3321be035/manager/0.log" Jul 09 14:53:09.172362 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:09.172328 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799fccbb74-28pmh_8d3c671f-941a-4f08-95bf-7102fcc80037/kube-auth-proxy/0.log" Jul 09 14:53:09.381530 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:09.381500 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-vjmxz_f8b67823-9b07-449f-a97a-547a3c2ac4c3/payload-pre-processing/0.log" Jul 09 14:53:09.440234 ip-10-0-142-0 kubenswrapper[2556]: I0709 14:53:09.440170 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-ms9cp_b072044b-0a70-4c8b-a3de-a8ecab3cf841/payload-processing/0.log"