Aug 01 00:27:08.333040 ip-10-0-140-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 01 00:27:08.333051 ip-10-0-140-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 01 00:27:08.333058 ip-10-0-140-111 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 01 00:27:08.333297 ip-10-0-140-111 systemd[1]: Failed to start Kubernetes Kubelet. Aug 01 00:27:18.410408 ip-10-0-140-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 01 00:27:18.410424 ip-10-0-140-111 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5f139b2febd44a03a343083dec019248 -- Aug 01 00:29:10.521459 ip-10-0-140-111 systemd[1]: Starting Kubernetes Kubelet... Aug 01 00:29:10.937289 ip-10-0-140-111 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:29:10.937289 ip-10-0-140-111 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 01 00:29:10.937289 ip-10-0-140-111 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:29:10.937289 ip-10-0-140-111 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 01 00:29:10.937289 ip-10-0-140-111 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 01 00:29:10.938392 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.938258 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 01 00:29:10.945007 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.944981 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:29:10.945007 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945002 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:29:10.945007 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945007 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:29:10.945007 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945011 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:29:10.945007 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945015 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945020 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945024 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945028 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945032 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945035 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945039 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945043 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945048 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945052 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945056 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945060 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945064 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945069 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945073 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945077 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945080 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945084 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945088 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:29:10.945336 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945092 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945097 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945101 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945105 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945110 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945114 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945118 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945122 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945127 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945130 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945135 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945139 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945143 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945148 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945165 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945172 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945176 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945181 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945185 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:29:10.946111 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945189 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945193 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945198 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945202 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945206 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945210 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945215 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945220 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945228 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945232 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945236 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945240 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945247 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945253 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945258 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945264 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945269 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945273 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945277 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945282 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:29:10.946901 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945286 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945290 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945294 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945299 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945303 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945307 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945312 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945316 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945321 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945330 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945336 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945341 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945346 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945353 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945358 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945362 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945367 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945371 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945376 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:29:10.947800 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945380 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945384 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945389 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945393 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945397 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945401 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945405 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945409 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945414 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945418 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945424 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945430 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945435 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945439 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945444 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945448 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945452 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.945457 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946752 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946763 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:29:10.948621 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946768 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946773 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946778 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946782 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946787 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946791 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946796 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946801 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946805 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946812 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946818 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946823 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946828 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946832 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946837 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946841 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946846 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946850 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946854 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946859 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:29:10.949486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946864 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946868 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946873 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946877 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946881 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946885 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946889 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946893 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946897 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946901 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946906 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946910 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946914 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946919 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946923 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946928 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946932 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946936 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946940 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:29:10.950241 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946947 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946955 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946959 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946964 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946969 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946974 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946980 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946984 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946989 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946993 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.946997 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947001 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947006 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947010 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947014 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947018 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947023 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947027 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947032 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947036 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:29:10.950811 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947041 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947046 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947050 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947055 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947059 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947066 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947070 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947075 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947079 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947084 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947088 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947092 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947097 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947101 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947108 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947112 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947116 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947121 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947125 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947129 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:29:10.951429 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947133 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947137 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947142 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947146 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947151 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947184 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947188 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947192 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947197 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947201 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947207 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947213 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947219 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947224 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947228 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947232 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947236 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947240 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947886 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947895 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:29:10.952144 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947900 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947904 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947909 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947913 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947917 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947921 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947926 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947931 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947935 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947940 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947944 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947949 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947953 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947957 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947961 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947965 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947969 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947974 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947978 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:29:10.952672 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947982 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947986 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947991 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947995 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.947999 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948004 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948008 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948012 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948019 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948025 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948029 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948034 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948038 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948043 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948047 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948051 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948055 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948060 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948064 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:29:10.953172 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948068 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948073 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948077 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948081 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948086 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948090 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948094 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948098 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948102 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948107 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948111 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948117 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948121 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948125 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948129 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948133 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948138 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948143 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948147 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948171 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:29:10.953642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948176 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948180 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948184 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948189 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948193 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948198 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948202 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948206 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948210 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948214 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948218 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948223 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948227 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948233 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948239 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948244 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948248 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948252 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948256 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948260 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:29:10.954230 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948265 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948270 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948274 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948280 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948286 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948290 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948295 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948299 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948304 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948308 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948312 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948316 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948322 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948325 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948329 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948334 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948348 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948353 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.948357 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948476 2575 flags.go:64] FLAG: --address="0.0.0.0" Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948488 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 01 00:29:10.954727 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948500 2575 flags.go:64] FLAG: --anonymous-auth="true" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948508 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948516 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948521 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948529 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948536 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948542 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948547 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948553 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948559 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948564 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948573 2575 flags.go:64] FLAG: --cgroup-root="" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948578 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948582 2575 flags.go:64] FLAG: --client-ca-file="" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948587 2575 flags.go:64] FLAG: --cloud-provider="external" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948592 2575 flags.go:64] FLAG: --cluster-dns="[]" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948599 2575 flags.go:64] FLAG: --cluster-domain="" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948604 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948609 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948614 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948619 2575 flags.go:64] FLAG: --container-log-max-files="5" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948625 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948630 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 01 00:29:10.955261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948636 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948641 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948646 2575 flags.go:64] FLAG: --contention-profiling="false" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948651 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948657 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948663 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948668 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948675 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948680 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948685 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948689 2575 flags.go:64] FLAG: --enable-load-reader="false" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948695 2575 flags.go:64] FLAG: --enable-server="true" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948700 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948708 2575 flags.go:64] FLAG: --event-burst="100" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948713 2575 flags.go:64] FLAG: --event-qps="50" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948718 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948724 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948729 2575 flags.go:64] FLAG: --eviction-hard="" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948735 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948740 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948746 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948751 2575 flags.go:64] FLAG: --eviction-soft="" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948757 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948761 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948766 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 01 00:29:10.955829 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948771 2575 flags.go:64] FLAG: --experimental-mounter-path="" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948776 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948781 2575 flags.go:64] FLAG: --fail-swap-on="true" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948785 2575 flags.go:64] FLAG: --feature-gates="" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948792 2575 flags.go:64] FLAG: --file-check-frequency="20s" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948797 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948802 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948807 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948812 2575 flags.go:64] FLAG: --healthz-port="10248" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948817 2575 flags.go:64] FLAG: --help="false" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948822 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-111.ec2.internal" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948829 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948834 2575 flags.go:64] FLAG: --http-check-frequency="20s" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948839 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948845 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948851 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948856 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948860 2575 flags.go:64] FLAG: --image-service-endpoint="" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948865 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948870 2575 flags.go:64] FLAG: --kube-api-burst="100" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948875 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948880 2575 flags.go:64] FLAG: --kube-api-qps="50" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948885 2575 flags.go:64] FLAG: --kube-reserved="" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948890 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 01 00:29:10.956499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948895 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948900 2575 flags.go:64] FLAG: --kubelet-cgroups="" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948905 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948911 2575 flags.go:64] FLAG: --lock-file="" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948916 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948921 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948926 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948935 2575 flags.go:64] FLAG: --log-json-split-stream="false" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948940 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948945 2575 flags.go:64] FLAG: --log-text-split-stream="false" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948949 2575 flags.go:64] FLAG: --logging-format="text" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948954 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948960 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948965 2575 flags.go:64] FLAG: --manifest-url="" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948969 2575 flags.go:64] FLAG: --manifest-url-header="" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948976 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948981 2575 flags.go:64] FLAG: --max-open-files="1000000" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948988 2575 flags.go:64] FLAG: --max-pods="110" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.948994 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949000 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949005 2575 flags.go:64] FLAG: --memory-manager-policy="None" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949010 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949015 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949020 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949024 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 01 00:29:10.957100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949038 2575 flags.go:64] FLAG: --node-status-max-images="50" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949043 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949048 2575 flags.go:64] FLAG: --oom-score-adj="-999" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949053 2575 flags.go:64] FLAG: --pod-cidr="" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949057 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949066 2575 flags.go:64] FLAG: --pod-manifest-path="" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949071 2575 flags.go:64] FLAG: --pod-max-pids="-1" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949076 2575 flags.go:64] FLAG: --pods-per-core="0" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949082 2575 flags.go:64] FLAG: --port="10250" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949087 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949091 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0834bd4090bc2dac8" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949098 2575 flags.go:64] FLAG: --qos-reserved="" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949103 2575 flags.go:64] FLAG: --read-only-port="10255" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949108 2575 flags.go:64] FLAG: --register-node="true" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949113 2575 flags.go:64] FLAG: --register-with-taints="" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949119 2575 flags.go:64] FLAG: --registry-burst="10" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949124 2575 flags.go:64] FLAG: --registry-qps="5" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949130 2575 flags.go:64] FLAG: --reserved-cpus="" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949134 2575 flags.go:64] FLAG: --reserved-memory="" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949140 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949145 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949150 2575 flags.go:64] FLAG: --rotate-certificates="false" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949173 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949178 2575 flags.go:64] FLAG: --runonce="false" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949183 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 01 00:29:10.957771 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949189 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949195 2575 flags.go:64] FLAG: --seccomp-default="false" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949200 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949205 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949210 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949215 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949220 2575 flags.go:64] FLAG: --storage-driver-password="root" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949225 2575 flags.go:64] FLAG: --storage-driver-secure="false" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949230 2575 flags.go:64] FLAG: --storage-driver-table="stats" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949235 2575 flags.go:64] FLAG: --storage-driver-user="root" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949240 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949245 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949250 2575 flags.go:64] FLAG: --system-cgroups="" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949255 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949265 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949270 2575 flags.go:64] FLAG: --tls-cert-file="" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949275 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949282 2575 flags.go:64] FLAG: --tls-min-version="" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949288 2575 flags.go:64] FLAG: --tls-private-key-file="" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949292 2575 flags.go:64] FLAG: --topology-manager-policy="none" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949298 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949303 2575 flags.go:64] FLAG: --topology-manager-scope="container" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949308 2575 flags.go:64] FLAG: --v="2" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949314 2575 flags.go:64] FLAG: --version="false" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949320 2575 flags.go:64] FLAG: --vmodule="" Aug 01 00:29:10.958525 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949327 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.949332 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949537 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949543 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949548 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949555 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949561 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949567 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949578 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949583 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949588 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949593 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949598 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949602 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949607 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949612 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949617 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949621 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949626 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:29:10.959166 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949631 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949635 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949640 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949644 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949648 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949654 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949661 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949665 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949670 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949674 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949678 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949683 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949688 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949692 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949696 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949700 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949705 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949709 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949713 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949717 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:29:10.959639 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949721 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949730 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949736 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949741 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949745 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949749 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949754 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949758 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949762 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949767 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949771 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949775 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949780 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949784 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949788 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949793 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949797 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949801 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949808 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:29:10.960132 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949815 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949821 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949825 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949830 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949834 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949839 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949843 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949847 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949851 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949855 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949860 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949864 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949868 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949873 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949878 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949882 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949887 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949891 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949895 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949899 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:29:10.960682 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949904 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949908 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949912 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949916 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949921 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949925 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949929 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949933 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949937 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949941 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949946 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949952 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949957 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949961 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949966 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949970 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949974 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949978 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949982 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949986 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:29:10.961198 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949990 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949995 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.949999 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.950567 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.957900 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.957916 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.957937 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.957943 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.957999 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958006 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958011 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958015 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958018 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958021 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:29:10.961701 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958024 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958027 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958030 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958033 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958036 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958039 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958043 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958047 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958050 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958053 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958055 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958058 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958061 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958064 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958067 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958070 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958072 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958075 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958079 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:29:10.962047 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958083 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958086 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958088 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958091 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958095 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958098 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958101 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958103 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958106 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958109 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958112 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958114 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958117 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958119 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958122 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958124 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958127 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958129 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958132 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958135 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:29:10.962526 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958137 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958140 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958142 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958145 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958147 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958150 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958152 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958172 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958174 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958177 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958180 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958183 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958185 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958188 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958191 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958193 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958198 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958201 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958204 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:29:10.963014 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958206 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958209 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958212 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958214 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958217 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958220 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958222 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958225 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958228 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958230 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958233 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958236 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958238 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958241 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958244 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958246 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958249 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958251 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958254 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958257 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958259 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:29:10.963486 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958262 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958264 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958267 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958269 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958272 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958275 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958277 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958280 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958283 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958286 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958289 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958292 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958295 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958297 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:29:10.963991 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.958303 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958420 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958425 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958428 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958431 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958434 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958436 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958440 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958442 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958445 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958449 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958453 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958456 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958460 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958463 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958466 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958468 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958471 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958473 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 01 00:29:10.964489 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958476 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958478 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958481 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958483 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958486 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958489 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958492 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958495 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958498 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958501 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958504 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958507 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958509 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958512 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958514 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958517 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958520 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958522 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958525 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958528 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 01 00:29:10.964951 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958531 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958534 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958536 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958539 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958542 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958545 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958548 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958551 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958553 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958556 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958558 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958561 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958563 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958566 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958568 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958571 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958574 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958576 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958579 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958582 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 01 00:29:10.965504 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958585 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958587 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958590 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958593 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958596 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958598 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958601 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958604 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958606 2575 feature_gate.go:328] unrecognized feature gate: Example2 Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958609 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958611 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958614 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958617 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958619 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958622 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958624 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958627 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958629 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958633 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958636 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 01 00:29:10.966358 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958639 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958643 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958645 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958648 2575 feature_gate.go:328] unrecognized feature gate: Example Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958651 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958654 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958657 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958659 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958662 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958665 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958667 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958670 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958673 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958675 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958678 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958680 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958683 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958686 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958688 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958691 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 01 00:29:10.967030 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:10.958694 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 01 00:29:10.967665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.958699 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 01 00:29:10.967665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.958832 2575 server.go:962] "Client rotation is on, will bootstrap in background" Aug 01 00:29:10.967665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.960874 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 01 00:29:10.967665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.962396 2575 server.go:1019] "Starting client certificate rotation" Aug 01 00:29:10.967665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.962498 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 01 00:29:10.967665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.962540 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 01 00:29:10.967665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.965010 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 01 00:29:10.968462 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.968440 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 01 00:29:10.979511 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.979487 2575 log.go:25] "Validated CRI v1 runtime API" Aug 01 00:29:10.981805 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.981785 2575 log.go:25] "Validated CRI v1 image API" Aug 01 00:29:10.983550 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.983531 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 01 00:29:10.986148 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.986123 2575 fs.go:135] Filesystem UUIDs: map[795a678a-01f2-4937-a5c3-3cbed80b8e16:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ea59784d-d096-431f-8f4f-6d02a71a857c:/dev/nvme0n1p3] Aug 01 00:29:10.986227 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.986147 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 01 00:29:10.991218 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.991075 2575 manager.go:216] Machine: {Timestamp:2026-08-01 00:29:10.9899647 +0000 UTC m=+0.346756347 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108730 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22a053943d411aeb729d72aee9a2f3 SystemUUID:ec22a053-943d-411a-eb72-9d72aee9a2f3 BootID:5f139b2f-ebd4-4a03-a343-083dec019248 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1c:ed:ec:62:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1c:ed:ec:62:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:c9:fe:50:58:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 01 00:29:10.991218 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.991208 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 01 00:29:10.991357 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.991302 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 01 00:29:10.992176 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.992137 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 01 00:29:10.995039 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.995007 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 01 00:29:10.995209 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.995043 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 01 00:29:10.995781 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.995770 2575 topology_manager.go:138] "Creating topology manager with none policy" Aug 01 00:29:10.995812 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.995784 2575 container_manager_linux.go:309] "Creating device plugin manager" Aug 01 00:29:10.995812 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.995796 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 01 00:29:10.996705 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.996693 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 01 00:29:10.997807 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:10.997796 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 01 00:29:11.001216 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001200 2575 state_mem.go:36] "Initialized new in-memory state store" Aug 01 00:29:11.001365 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001355 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 01 00:29:11.001394 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001389 2575 kubelet.go:486] "Attempting to sync node with API server" Aug 01 00:29:11.001430 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001414 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 01 00:29:11.001462 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001444 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 01 00:29:11.001462 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001460 2575 kubelet.go:398] "Adding apiserver pod source" Aug 01 00:29:11.001521 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001472 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 01 00:29:11.001953 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001941 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 01 00:29:11.002014 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.001962 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 01 00:29:11.004187 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.004168 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 01 00:29:11.005796 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.005779 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 01 00:29:11.005838 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.005814 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 01 00:29:11.007140 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007121 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007147 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007173 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007184 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007199 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007216 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007229 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007238 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007250 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007259 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007271 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 01 00:29:11.007295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.007280 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 01 00:29:11.008461 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.008064 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 01 00:29:11.008461 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.008088 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 01 00:29:11.011592 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.011576 2575 server.go:1268] "Started kubelet" Aug 01 00:29:11.011717 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.011660 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 01 00:29:11.011775 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.011755 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 01 00:29:11.011843 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.011664 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 01 00:29:11.012101 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.012079 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 01 00:29:11.012462 ip-10-0-140-111 systemd[1]: Started Kubernetes Kubelet. Aug 01 00:29:11.013064 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.013000 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 01 00:29:11.013127 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.013073 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 01 00:29:11.013127 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.013103 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 01 00:29:11.013965 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.013952 2575 server.go:310] "Adding debug handlers to kubelet server" Aug 01 00:29:11.017175 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.017129 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 01 00:29:11.018547 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.017570 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-111.ec2.internal.18c786f1f3d1b4d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-111.ec2.internal,UID:ip-10-0-140-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-111.ec2.internal,},FirstTimestamp:2026-08-01 00:29:11.011538131 +0000 UTC m=+0.368329778,LastTimestamp:2026-08-01 00:29:11.011538131 +0000 UTC m=+0.368329778,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-111.ec2.internal,}" Aug 01 00:29:11.018547 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.018547 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 01 00:29:11.018724 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.018558 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 01 00:29:11.018724 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.018623 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.019274 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019255 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 01 00:29:11.019274 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019276 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 01 00:29:11.019419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019259 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 01 00:29:11.019419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019395 2575 reconstruct.go:97] "Volume reconstruction finished" Aug 01 00:29:11.019419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019402 2575 reconciler.go:29] "Reconciler: start to sync state" Aug 01 00:29:11.019620 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019602 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jnh24" Aug 01 00:29:11.019676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019630 2575 factory.go:153] Registering CRI-O factory Aug 01 00:29:11.019676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019661 2575 factory.go:223] Registration of the crio container factory successfully Aug 01 00:29:11.019758 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019703 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 01 00:29:11.019758 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019711 2575 factory.go:55] Registering systemd factory Aug 01 00:29:11.019758 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019716 2575 factory.go:223] Registration of the systemd container factory successfully Aug 01 00:29:11.019758 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019737 2575 factory.go:103] Registering Raw factory Aug 01 00:29:11.019758 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.019746 2575 manager.go:1196] Started watching for new ooms in manager Aug 01 00:29:11.020083 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.020072 2575 manager.go:349] Starting recovery of all containers Aug 01 00:29:11.026504 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.026473 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 01 00:29:11.027269 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.027243 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jnh24" Aug 01 00:29:11.027450 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.027430 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 01 00:29:11.030265 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.030079 2575 manager.go:354] Recovery completed Aug 01 00:29:11.036416 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.036397 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:29:11.038146 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.038131 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:29:11.038214 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.038176 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:29:11.038214 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.038191 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:29:11.038749 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.038728 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 01 00:29:11.038749 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.038739 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:29:11.038749 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.038744 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 01 00:29:11.038911 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.038763 2575 state_mem.go:36] "Initialized new in-memory state store" Aug 01 00:29:11.039380 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.039363 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:29:11.039459 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.039396 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:29:11.039459 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.039411 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:29:11.039894 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.039880 2575 policy_none.go:49] "None policy: Start" Aug 01 00:29:11.039953 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.039900 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 01 00:29:11.039953 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.039913 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 01 00:29:11.040852 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.040842 2575 policy_none.go:47] "Start" Aug 01 00:29:11.072097 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.071882 2575 manager.go:337] "Starting Device Plugin manager" Aug 01 00:29:11.072242 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.072130 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 01 00:29:11.072242 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.072143 2575 server.go:87] "Starting device plugin registration server" Aug 01 00:29:11.072413 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.072400 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 01 00:29:11.072464 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.072416 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 01 00:29:11.072515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.072498 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 01 00:29:11.072595 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.072585 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 01 00:29:11.072647 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.072596 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 01 00:29:11.073172 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.073135 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 01 00:29:11.073280 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.073214 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.155633 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.155591 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 01 00:29:11.156807 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.156791 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 01 00:29:11.156907 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.156834 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 01 00:29:11.156907 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.156861 2575 kubelet.go:2448] "Starting kubelet main sync loop" Aug 01 00:29:11.156996 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.156904 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 01 00:29:11.159100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.159080 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:29:11.172947 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.172920 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:29:11.173905 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.173883 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:29:11.174011 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.173920 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:29:11.174011 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.173931 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:29:11.174011 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.173954 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.183012 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.182988 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.183093 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.183017 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-111.ec2.internal\": node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.196896 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.196791 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.257686 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.257632 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal"] Aug 01 00:29:11.257848 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.257760 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:29:11.258825 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.258809 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:29:11.258923 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.258842 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:29:11.258923 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.258856 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:29:11.260141 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.260125 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:29:11.260268 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.260253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.260324 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.260282 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:29:11.260923 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.260909 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:29:11.260989 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.260936 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:29:11.260989 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.260948 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:29:11.261725 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.261711 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:29:11.261778 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.261735 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:29:11.261778 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.261745 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:29:11.262128 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.262114 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.262183 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.262168 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 01 00:29:11.262820 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.262804 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientMemory" Aug 01 00:29:11.262900 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.262833 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasNoDiskPressure" Aug 01 00:29:11.262900 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.262846 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeHasSufficientPID" Aug 01 00:29:11.287033 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.287000 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.291789 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.291767 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-111.ec2.internal\" not found" node="ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.297487 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.297467 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.321701 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.321671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/926883367cca128b6b84390dda10a55b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"926883367cca128b6b84390dda10a55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.321836 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.321705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926883367cca128b6b84390dda10a55b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"926883367cca128b6b84390dda10a55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.321836 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.321730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c636173225f22081c0d5c73d018d010-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"7c636173225f22081c0d5c73d018d010\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.397832 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.397785 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.422283 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.422254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926883367cca128b6b84390dda10a55b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"926883367cca128b6b84390dda10a55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.422391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.422289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c636173225f22081c0d5c73d018d010-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"7c636173225f22081c0d5c73d018d010\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.422391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.422306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/926883367cca128b6b84390dda10a55b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"926883367cca128b6b84390dda10a55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.422391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.422361 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/926883367cca128b6b84390dda10a55b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"926883367cca128b6b84390dda10a55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.422515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.422384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/926883367cca128b6b84390dda10a55b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal\" (UID: \"926883367cca128b6b84390dda10a55b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.422515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.422366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c636173225f22081c0d5c73d018d010-config\") pod \"kube-apiserver-proxy-ip-10-0-140-111.ec2.internal\" (UID: \"7c636173225f22081c0d5c73d018d010\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.498610 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.498533 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.590327 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.590296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.594994 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.594485 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Aug 01 00:29:11.599137 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.599121 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.699765 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.699717 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.800405 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.800317 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.900885 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:11.900842 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:11.961427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.961395 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 01 00:29:11.961900 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.961546 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 01 00:29:11.994102 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:11.994073 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:29:12.001211 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:12.001188 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:12.035367 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.035320 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-31 00:24:11 +0000 UTC" deadline="2028-02-14 04:13:43.032462681 +0000 UTC" Aug 01 00:29:12.035367 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.035357 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13491h44m30.997108792s" Aug 01 00:29:12.101685 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:12.101619 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:12.138975 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:12.138939 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926883367cca128b6b84390dda10a55b.slice/crio-ac09fbd17af4e3318c366f1110d8123e8a63371708d85bcde694912ca9e6915e WatchSource:0}: Error finding container ac09fbd17af4e3318c366f1110d8123e8a63371708d85bcde694912ca9e6915e: Status 404 returned error can't find the container with id ac09fbd17af4e3318c366f1110d8123e8a63371708d85bcde694912ca9e6915e Aug 01 00:29:12.139323 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:12.139298 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c636173225f22081c0d5c73d018d010.slice/crio-74dcd3507a683723f7706dcd15ea7b39b266b408384a86514f2678fe446056be WatchSource:0}: Error finding container 74dcd3507a683723f7706dcd15ea7b39b266b408384a86514f2678fe446056be: Status 404 returned error can't find the container with id 74dcd3507a683723f7706dcd15ea7b39b266b408384a86514f2678fe446056be Aug 01 00:29:12.143332 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.143315 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:29:12.160413 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.160357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"926883367cca128b6b84390dda10a55b","Type":"ContainerStarted","Data":"ac09fbd17af4e3318c366f1110d8123e8a63371708d85bcde694912ca9e6915e"} Aug 01 00:29:12.161446 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.161425 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" event={"ID":"7c636173225f22081c0d5c73d018d010","Type":"ContainerStarted","Data":"74dcd3507a683723f7706dcd15ea7b39b266b408384a86514f2678fe446056be"} Aug 01 00:29:12.202694 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:12.202668 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:12.303240 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:12.303201 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:12.403780 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:12.403701 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:12.500044 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.500011 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:29:12.504207 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:12.504186 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-111.ec2.internal\" not found" Aug 01 00:29:12.563301 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.563275 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:29:12.619277 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.619248 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" Aug 01 00:29:12.631576 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.631539 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 01 00:29:12.632726 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.632703 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" Aug 01 00:29:12.642583 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:12.642545 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 01 00:29:13.003279 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.003175 2575 apiserver.go:52] "Watching apiserver" Aug 01 00:29:13.012032 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.011984 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 01 00:29:13.013480 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.013445 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-kmmsg","openshift-ovn-kubernetes/ovnkube-node-rt596","kube-system/global-pull-secret-syncer-gkwsd","kube-system/konnectivity-agent-l7qqq","kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8h82v","openshift-dns/node-resolver-xpfmg","openshift-image-registry/node-ca-tv49p","openshift-multus/multus-additional-cni-plugins-b7zfn","openshift-network-diagnostics/network-check-target-p9lqk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal","openshift-multus/multus-zn797","openshift-multus/network-metrics-daemon-kkp8g"] Aug 01 00:29:13.015461 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.015435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.016790 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.016769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.018012 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.017933 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x6dbx\"" Aug 01 00:29:13.018012 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.017982 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 01 00:29:13.018208 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.018100 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 01 00:29:13.018437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.018422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.018526 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.018487 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:13.019094 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.019071 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 01 00:29:13.019191 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.019108 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 01 00:29:13.019362 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.019345 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 01 00:29:13.019488 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.019467 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 01 00:29:13.019557 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.019514 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp8hn\"" Aug 01 00:29:13.020166 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.020146 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.021905 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.021883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:13.021994 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.021963 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:13.022551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.022528 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mz2cv\"" Aug 01 00:29:13.022651 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.022565 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 01 00:29:13.022866 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.022849 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 01 00:29:13.023417 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.023396 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.025158 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.025130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.025780 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.025757 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:29:13.025892 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.025815 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mbxgq\"" Aug 01 00:29:13.026040 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.026024 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 01 00:29:13.026564 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.026543 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn797" Aug 01 00:29:13.027592 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.027569 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z2nt4\"" Aug 01 00:29:13.027977 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.027962 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 01 00:29:13.028089 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.028011 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 01 00:29:13.028089 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.028038 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 01 00:29:13.028304 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.028284 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 01 00:29:13.028384 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.028318 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 01 00:29:13.028384 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.028368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.029081 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.029059 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4bhtw\"" Aug 01 00:29:13.029425 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.029406 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 01 00:29:13.031371 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.030622 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:13.031371 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.030699 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:13.031371 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031069 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031746 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-registration-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-device-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-dbus\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e315e32-14a6-4ca3-8015-3593e3d3026c-agent-certs\") pod \"konnectivity-agent-l7qqq\" (UID: \"5e315e32-14a6-4ca3-8015-3593e3d3026c\") " pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.031978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-tmp-dir\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-sys-fs\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e315e32-14a6-4ca3-8015-3593e3d3026c-konnectivity-ca\") pod \"konnectivity-agent-l7qqq\" (UID: \"5e315e32-14a6-4ca3-8015-3593e3d3026c\") " pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-hosts-file\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032152 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-socket-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sjnm\" (UniqueName: \"kubernetes.io/projected/0e31ca4c-eb13-480c-83dd-f8d1988c736a-kube-api-access-4sjnm\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-kubelet-config\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.032904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.033875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82kbn\" (UniqueName: \"kubernetes.io/projected/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-kube-api-access-82kbn\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.033875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032555 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 01 00:29:13.033875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032631 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 01 00:29:13.033875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032645 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r9tnq\"" Aug 01 00:29:13.033875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.032821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.033875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.033147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.036150 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.035863 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8gqb\"" Aug 01 00:29:13.036150 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.035944 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 01 00:29:13.036150 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.036033 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:29:13.036447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.036223 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 01 00:29:13.036447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.036253 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 01 00:29:13.036772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.036566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dv5t2\"" Aug 01 00:29:13.036772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.036613 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 01 00:29:13.036772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.036645 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 01 00:29:13.057398 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.057360 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-46pdl" Aug 01 00:29:13.065807 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.065775 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-46pdl" Aug 01 00:29:13.120951 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.120918 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 01 00:29:13.133466 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e315e32-14a6-4ca3-8015-3593e3d3026c-konnectivity-ca\") pod \"konnectivity-agent-l7qqq\" (UID: \"5e315e32-14a6-4ca3-8015-3593e3d3026c\") " pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.133466 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-hosts-file\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysctl-d\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-tmp\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-kubelet\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-hosts-file\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sjnm\" (UniqueName: \"kubernetes.io/projected/0e31ca4c-eb13-480c-83dd-f8d1988c736a-kube-api-access-4sjnm\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysconfig\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.133735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-kubernetes\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cnibin\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-socket-dir-parent\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-k8s-cni-cncf-io\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-slash\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-systemd\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03487da-17ad-4def-9420-a85d74d06f0b-cni-binary-copy\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-netns\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.133994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-device-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e315e32-14a6-4ca3-8015-3593e3d3026c-agent-certs\") pod \"konnectivity-agent-l7qqq\" (UID: \"5e315e32-14a6-4ca3-8015-3593e3d3026c\") " pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134070 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-device-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134079 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.134152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-host\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xst4l\" (UniqueName: \"kubernetes.io/projected/2b480d6a-2902-469c-9b07-401b5826775a-kube-api-access-xst4l\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-system-cni-dir\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134306 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e315e32-14a6-4ca3-8015-3593e3d3026c-konnectivity-ca\") pod \"konnectivity-agent-l7qqq\" (UID: \"5e315e32-14a6-4ca3-8015-3593e3d3026c\") " pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-systemd\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-log-socket\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-cni-bin\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134457 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9tb\" (UniqueName: \"kubernetes.io/projected/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-kube-api-access-rw9tb\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-cni-multus\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-run-netns\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-etc-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.134800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134611 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-node-log\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovnkube-config\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-kubelet-config\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysctl-conf\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134709 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-kubelet-config\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-cnibin\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-hostroot\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llrx\" (UniqueName: \"kubernetes.io/projected/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-kube-api-access-9llrx\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-registration-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-dbus\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-tmp-dir\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-registration-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-sys\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.134989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-cni-bin\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-dbus\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.135546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-env-overrides\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-multus-certs\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovnkube-script-lib\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvm8c\" (UniqueName: \"kubernetes.io/projected/84fd6a55-cd43-4cfd-8750-41c249783dd8-kube-api-access-mvm8c\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-tmp-dir\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-tuned\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-cni-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-etc-kubernetes\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-socket-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82kbn\" (UniqueName: \"kubernetes.io/projected/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-kube-api-access-82kbn\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-os-release\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.135422 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-kubelet\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-socket-dir\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.136427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljcqg\" (UniqueName: \"kubernetes.io/projected/c03487da-17ad-4def-9420-a85d74d06f0b-kube-api-access-ljcqg\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.135501 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret podName:6e26feb7-e8a1-43f0-a20d-a18db3aa016b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:13.635479865 +0000 UTC m=+2.992271526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret") pod "global-pull-secret-syncer-gkwsd" (UID: "6e26feb7-e8a1-43f0-a20d-a18db3aa016b") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-var-lib-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-conf-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-modprobe-d\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-system-cni-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03487da-17ad-4def-9420-a85d74d06f0b-multus-daemon-config\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-systemd-units\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-sys-fs\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-var-lib-kubelet\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135797 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e31ca4c-eb13-480c-83dd-f8d1988c736a-sys-fs\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovn-node-metrics-cert\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b480d6a-2902-469c-9b07-401b5826775a-host-slash\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-os-release\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-host\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.137131 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-run\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjsls\" (UniqueName: \"kubernetes.io/projected/bf6cd5d5-6fee-4e49-bda7-30a29a317638-kube-api-access-sjsls\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.135974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-serviceca\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.136000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-lib-modules\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.136025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b480d6a-2902-469c-9b07-401b5826775a-iptables-alerter-script\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.136052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tffsl\" (UniqueName: \"kubernetes.io/projected/171e54b7-3ce9-4b0d-96cc-477bdac0a271-kube-api-access-tffsl\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.136076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-ovn\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.136124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-run-ovn-kubernetes\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.138007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.136158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-cni-netd\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.138510 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.138488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e315e32-14a6-4ca3-8015-3593e3d3026c-agent-certs\") pod \"konnectivity-agent-l7qqq\" (UID: \"5e315e32-14a6-4ca3-8015-3593e3d3026c\") " pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.143062 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.143030 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sjnm\" (UniqueName: \"kubernetes.io/projected/0e31ca4c-eb13-480c-83dd-f8d1988c736a-kube-api-access-4sjnm\") pod \"aws-ebs-csi-driver-node-qhtq8\" (UID: \"0e31ca4c-eb13-480c-83dd-f8d1988c736a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.144326 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.144300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82kbn\" (UniqueName: \"kubernetes.io/projected/7aa2e9a2-970b-41e5-a04a-9b5b75b5b499-kube-api-access-82kbn\") pod \"node-resolver-xpfmg\" (UID: \"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499\") " pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.236864 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.236823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-run-netns\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.236874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-etc-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.236903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.236923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-node-log\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.236931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-run-netns\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.236948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovnkube-config\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.236992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-etc-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysctl-conf\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-cnibin\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-hostroot\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llrx\" (UniqueName: \"kubernetes.io/projected/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-kube-api-access-9llrx\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-sys\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-cni-bin\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237482 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-hostroot\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-env-overrides\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-multus-certs\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-multus-certs\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovnkube-script-lib\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvm8c\" (UniqueName: \"kubernetes.io/projected/84fd6a55-cd43-4cfd-8750-41c249783dd8-kube-api-access-mvm8c\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-tuned\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovnkube-config\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-cni-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-cnibin\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.237896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-etc-kubernetes\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.237901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysctl-conf\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238103 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-cni-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovnkube-script-lib\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf6cd5d5-6fee-4e49-bda7-30a29a317638-env-overrides\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-cni-bin\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-node-log\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238377 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-sys\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-os-release\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-kubelet\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238481 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-os-release\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-etc-kubernetes\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-kubelet\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcqg\" (UniqueName: \"kubernetes.io/projected/c03487da-17ad-4def-9420-a85d74d06f0b-kube-api-access-ljcqg\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-var-lib-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-var-lib-openvswitch\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.238676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-conf-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-conf-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-modprobe-d\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-system-cni-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03487da-17ad-4def-9420-a85d74d06f0b-multus-daemon-config\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-systemd-units\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-var-lib-kubelet\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-system-cni-dir\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-modprobe-d\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-systemd-units\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovn-node-metrics-cert\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b480d6a-2902-469c-9b07-401b5826775a-host-slash\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.238972 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-var-lib-kubelet\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-os-release\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239058 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b480d6a-2902-469c-9b07-401b5826775a-host-slash\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-os-release\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-host\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.239574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239128 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-host\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-run\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjsls\" (UniqueName: \"kubernetes.io/projected/bf6cd5d5-6fee-4e49-bda7-30a29a317638-kube-api-access-sjsls\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239208 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-run\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-serviceca\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-lib-modules\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b480d6a-2902-469c-9b07-401b5826775a-iptables-alerter-script\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tffsl\" (UniqueName: \"kubernetes.io/projected/171e54b7-3ce9-4b0d-96cc-477bdac0a271-kube-api-access-tffsl\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-ovn\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-run-ovn-kubernetes\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-cni-netd\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c03487da-17ad-4def-9420-a85d74d06f0b-multus-daemon-config\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysctl-d\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-tmp\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-ovn\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-run-ovn-kubernetes\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.240534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-kubelet\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239544 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-cni-netd\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-kubelet\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-serviceca\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-lib-modules\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysconfig\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-kubernetes\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cnibin\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysconfig\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.239765 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239810 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-kubernetes\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-socket-dir-parent\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.239825 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs podName:84fd6a55-cd43-4cfd-8750-41c249783dd8 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:13.739805983 +0000 UTC m=+3.096597626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs") pod "network-metrics-daemon-kkp8g" (UID: "84fd6a55-cd43-4cfd-8750-41c249783dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-multus-socket-dir-parent\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cnibin\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-k8s-cni-cncf-io\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.241496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-slash\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-k8s-cni-cncf-io\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-systemd\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03487da-17ad-4def-9420-a85d74d06f0b-cni-binary-copy\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239956 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-slash\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.239986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-netns\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-host\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240029 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b480d6a-2902-469c-9b07-401b5826775a-iptables-alerter-script\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xst4l\" (UniqueName: \"kubernetes.io/projected/2b480d6a-2902-469c-9b07-401b5826775a-kube-api-access-xst4l\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-run-systemd\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-system-cni-dir\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-run-netns\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/171e54b7-3ce9-4b0d-96cc-477bdac0a271-system-cni-dir\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-systemd\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240145 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-host\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-log-socket\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-cni-bin\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.242405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-systemd\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-log-socket\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240306 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf6cd5d5-6fee-4e49-bda7-30a29a317638-host-cni-bin\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9tb\" (UniqueName: \"kubernetes.io/projected/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-kube-api-access-rw9tb\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-cni-multus\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240441 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c03487da-17ad-4def-9420-a85d74d06f0b-cni-binary-copy\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240497 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c03487da-17ad-4def-9420-a85d74d06f0b-host-var-lib-cni-multus\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240642 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-sysctl-d\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.240712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/171e54b7-3ce9-4b0d-96cc-477bdac0a271-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.242496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-tmp\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.243455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.243061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf6cd5d5-6fee-4e49-bda7-30a29a317638-ovn-node-metrics-cert\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.245175 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.245149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-etc-tuned\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.247716 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.247484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llrx\" (UniqueName: \"kubernetes.io/projected/131e1d80-8e9d-4665-82f5-b0e3eefeca8c-kube-api-access-9llrx\") pod \"node-ca-tv49p\" (UID: \"131e1d80-8e9d-4665-82f5-b0e3eefeca8c\") " pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.247716 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.247704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvm8c\" (UniqueName: \"kubernetes.io/projected/84fd6a55-cd43-4cfd-8750-41c249783dd8-kube-api-access-mvm8c\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:13.247891 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.247761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcqg\" (UniqueName: \"kubernetes.io/projected/c03487da-17ad-4def-9420-a85d74d06f0b-kube-api-access-ljcqg\") pod \"multus-zn797\" (UID: \"c03487da-17ad-4def-9420-a85d74d06f0b\") " pod="openshift-multus/multus-zn797" Aug 01 00:29:13.248717 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.248692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjsls\" (UniqueName: \"kubernetes.io/projected/bf6cd5d5-6fee-4e49-bda7-30a29a317638-kube-api-access-sjsls\") pod \"ovnkube-node-rt596\" (UID: \"bf6cd5d5-6fee-4e49-bda7-30a29a317638\") " pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.250544 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.250517 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:13.250649 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.250549 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:13.250649 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.250564 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jmnwl for pod openshift-network-diagnostics/network-check-target-p9lqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:13.250649 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.250644 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl podName:234cbdba-5336-48c0-9814-fcc0cdad9272 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:13.750624252 +0000 UTC m=+3.107415911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jmnwl" (UniqueName: "kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl") pod "network-check-target-p9lqk" (UID: "234cbdba-5336-48c0-9814-fcc0cdad9272") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:13.251960 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.251936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xst4l\" (UniqueName: \"kubernetes.io/projected/2b480d6a-2902-469c-9b07-401b5826775a-kube-api-access-xst4l\") pod \"iptables-alerter-kmmsg\" (UID: \"2b480d6a-2902-469c-9b07-401b5826775a\") " pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.253170 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.253136 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9tb\" (UniqueName: \"kubernetes.io/projected/a16738b4-f53f-4150-a8e1-1f14a0abb0b1-kube-api-access-rw9tb\") pod \"tuned-8h82v\" (UID: \"a16738b4-f53f-4150-a8e1-1f14a0abb0b1\") " pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.253323 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.253223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tffsl\" (UniqueName: \"kubernetes.io/projected/171e54b7-3ce9-4b0d-96cc-477bdac0a271-kube-api-access-tffsl\") pod \"multus-additional-cni-plugins-b7zfn\" (UID: \"171e54b7-3ce9-4b0d-96cc-477bdac0a271\") " pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.334456 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.334415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xpfmg" Aug 01 00:29:13.344278 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.344238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" Aug 01 00:29:13.354342 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.354314 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:13.361311 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.361287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8h82v" Aug 01 00:29:13.370243 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.370204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" Aug 01 00:29:13.379468 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.379439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zn797" Aug 01 00:29:13.389571 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.389546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:13.398707 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.398678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kmmsg" Aug 01 00:29:13.405900 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.405875 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tv49p" Aug 01 00:29:13.526030 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.525937 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 01 00:29:13.643991 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.643949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:13.644227 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.644131 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:13.644298 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.644231 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret podName:6e26feb7-e8a1-43f0-a20d-a18db3aa016b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:14.644209599 +0000 UTC m=+4.001001248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret") pod "global-pull-secret-syncer-gkwsd" (UID: "6e26feb7-e8a1-43f0-a20d-a18db3aa016b") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:13.744658 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.744621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:13.744816 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.744748 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:13.744816 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.744811 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs podName:84fd6a55-cd43-4cfd-8750-41c249783dd8 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:14.744793429 +0000 UTC m=+4.101585069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs") pod "network-metrics-daemon-kkp8g" (UID: "84fd6a55-cd43-4cfd-8750-41c249783dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:13.845321 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:13.845221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:13.845512 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.845408 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:13.845512 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.845432 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:13.845512 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.845445 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jmnwl for pod openshift-network-diagnostics/network-check-target-p9lqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:13.845669 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:13.845514 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl podName:234cbdba-5336-48c0-9814-fcc0cdad9272 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:14.845494519 +0000 UTC m=+4.202286169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmnwl" (UniqueName: "kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl") pod "network-check-target-p9lqk" (UID: "234cbdba-5336-48c0-9814-fcc0cdad9272") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:14.068645 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:14.068565 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:24:13 +0000 UTC" deadline="2028-03-10 19:25:27.694257879 +0000 UTC" Aug 01 00:29:14.068645 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:14.068644 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14106h56m13.62562415s" Aug 01 00:29:14.158142 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:14.157996 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:14.158363 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.158158 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:14.426307 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:14.426275 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6cd5d5_6fee_4e49_bda7_30a29a317638.slice/crio-f82dcf5aeac6d7d55331d0b6e83e258d97a838207c8afd2f9d754d4cc310fafd WatchSource:0}: Error finding container f82dcf5aeac6d7d55331d0b6e83e258d97a838207c8afd2f9d754d4cc310fafd: Status 404 returned error can't find the container with id f82dcf5aeac6d7d55331d0b6e83e258d97a838207c8afd2f9d754d4cc310fafd Aug 01 00:29:14.429388 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:14.429344 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e31ca4c_eb13_480c_83dd_f8d1988c736a.slice/crio-8cec4fb71560b544371eea5485fa5a1ab7866c65db43a259a51bb0a5dfc28e77 WatchSource:0}: Error finding container 8cec4fb71560b544371eea5485fa5a1ab7866c65db43a259a51bb0a5dfc28e77: Status 404 returned error can't find the container with id 8cec4fb71560b544371eea5485fa5a1ab7866c65db43a259a51bb0a5dfc28e77 Aug 01 00:29:14.432193 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:14.432145 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16738b4_f53f_4150_a8e1_1f14a0abb0b1.slice/crio-026c655b6b106e2b09ef8179367f606546a3f3f0b3ba0f4da4c6a445e36050fb WatchSource:0}: Error finding container 026c655b6b106e2b09ef8179367f606546a3f3f0b3ba0f4da4c6a445e36050fb: Status 404 returned error can't find the container with id 026c655b6b106e2b09ef8179367f606546a3f3f0b3ba0f4da4c6a445e36050fb Aug 01 00:29:14.433729 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:14.433698 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b480d6a_2902_469c_9b07_401b5826775a.slice/crio-1226da767bbbca7ead6ff2e220f86633f7b9e2ecf66a8649e952a887383b5942 WatchSource:0}: Error finding container 1226da767bbbca7ead6ff2e220f86633f7b9e2ecf66a8649e952a887383b5942: Status 404 returned error can't find the container with id 1226da767bbbca7ead6ff2e220f86633f7b9e2ecf66a8649e952a887383b5942 Aug 01 00:29:14.434758 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:14.434731 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa2e9a2_970b_41e5_a04a_9b5b75b5b499.slice/crio-73d8298902ea04955214b1209a2e0e785a27a26946ff0feb35caea99bff4af96 WatchSource:0}: Error finding container 73d8298902ea04955214b1209a2e0e785a27a26946ff0feb35caea99bff4af96: Status 404 returned error can't find the container with id 73d8298902ea04955214b1209a2e0e785a27a26946ff0feb35caea99bff4af96 Aug 01 00:29:14.435977 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:14.435946 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e315e32_14a6_4ca3_8015_3593e3d3026c.slice/crio-4da5d6e9a63a3d321b8f659fbbc037c6ce20ce4d56f74130a8909d0b8ba72fc5 WatchSource:0}: Error finding container 4da5d6e9a63a3d321b8f659fbbc037c6ce20ce4d56f74130a8909d0b8ba72fc5: Status 404 returned error can't find the container with id 4da5d6e9a63a3d321b8f659fbbc037c6ce20ce4d56f74130a8909d0b8ba72fc5 Aug 01 00:29:14.437243 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:14.436764 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03487da_17ad_4def_9420_a85d74d06f0b.slice/crio-18521d46b1df809fb92634ea59ee2720a09671a92d64336d7a9e2c1a6e5c1cd0 WatchSource:0}: Error finding container 18521d46b1df809fb92634ea59ee2720a09671a92d64336d7a9e2c1a6e5c1cd0: Status 404 returned error can't find the container with id 18521d46b1df809fb92634ea59ee2720a09671a92d64336d7a9e2c1a6e5c1cd0 Aug 01 00:29:14.653163 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:14.652934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:14.653306 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.653091 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:14.653306 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.653253 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret podName:6e26feb7-e8a1-43f0-a20d-a18db3aa016b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:16.653233608 +0000 UTC m=+6.010025248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret") pod "global-pull-secret-syncer-gkwsd" (UID: "6e26feb7-e8a1-43f0-a20d-a18db3aa016b") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:14.754445 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:14.754411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:14.754613 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.754526 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:14.754613 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.754576 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs podName:84fd6a55-cd43-4cfd-8750-41c249783dd8 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:16.7545632 +0000 UTC m=+6.111354834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs") pod "network-metrics-daemon-kkp8g" (UID: "84fd6a55-cd43-4cfd-8750-41c249783dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:14.855283 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:14.855247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:14.855508 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.855478 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:14.855508 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.855508 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:14.855649 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.855522 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jmnwl for pod openshift-network-diagnostics/network-check-target-p9lqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:14.855649 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:14.855584 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl podName:234cbdba-5336-48c0-9814-fcc0cdad9272 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:16.855563766 +0000 UTC m=+6.212355425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmnwl" (UniqueName: "kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl") pod "network-check-target-p9lqk" (UID: "234cbdba-5336-48c0-9814-fcc0cdad9272") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:15.068986 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.068919 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-31 00:24:13 +0000 UTC" deadline="2028-03-08 06:58:12.469082368 +0000 UTC" Aug 01 00:29:15.068986 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.068982 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14046h28m57.400104605s" Aug 01 00:29:15.157956 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.157255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:15.157956 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:15.157401 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:15.157956 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.157255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:15.157956 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:15.157903 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:15.179599 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.179497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerStarted","Data":"441284178af6d4d7ebe4513b7b526fd7ebe06cff61d1f10f5105baf4f3feed5c"} Aug 01 00:29:15.185504 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.185465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tv49p" event={"ID":"131e1d80-8e9d-4665-82f5-b0e3eefeca8c","Type":"ContainerStarted","Data":"fc08a38b787b3d653afb4d87f3840e67f50fa539e44ed2b153d403b6641d158c"} Aug 01 00:29:15.187281 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.187246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7qqq" event={"ID":"5e315e32-14a6-4ca3-8015-3593e3d3026c","Type":"ContainerStarted","Data":"4da5d6e9a63a3d321b8f659fbbc037c6ce20ce4d56f74130a8909d0b8ba72fc5"} Aug 01 00:29:15.190352 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.190259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpfmg" event={"ID":"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499","Type":"ContainerStarted","Data":"73d8298902ea04955214b1209a2e0e785a27a26946ff0feb35caea99bff4af96"} Aug 01 00:29:15.197006 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.196895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8h82v" event={"ID":"a16738b4-f53f-4150-a8e1-1f14a0abb0b1","Type":"ContainerStarted","Data":"026c655b6b106e2b09ef8179367f606546a3f3f0b3ba0f4da4c6a445e36050fb"} Aug 01 00:29:15.202753 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.202720 2575 generic.go:356] "Generic (PLEG): container finished" podID="926883367cca128b6b84390dda10a55b" containerID="1b1b80e892798cbd4dec2554d9362b3bbb3219e3e16baca11d49d379711decf5" exitCode=0 Aug 01 00:29:15.203460 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.203432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"926883367cca128b6b84390dda10a55b","Type":"ContainerDied","Data":"1b1b80e892798cbd4dec2554d9362b3bbb3219e3e16baca11d49d379711decf5"} Aug 01 00:29:15.208998 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.208838 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn797" event={"ID":"c03487da-17ad-4def-9420-a85d74d06f0b","Type":"ContainerStarted","Data":"18521d46b1df809fb92634ea59ee2720a09671a92d64336d7a9e2c1a6e5c1cd0"} Aug 01 00:29:15.212435 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.212407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" event={"ID":"0e31ca4c-eb13-480c-83dd-f8d1988c736a","Type":"ContainerStarted","Data":"8cec4fb71560b544371eea5485fa5a1ab7866c65db43a259a51bb0a5dfc28e77"} Aug 01 00:29:15.214884 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.214825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"f82dcf5aeac6d7d55331d0b6e83e258d97a838207c8afd2f9d754d4cc310fafd"} Aug 01 00:29:15.219546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.219519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" event={"ID":"7c636173225f22081c0d5c73d018d010","Type":"ContainerStarted","Data":"2c527f40c4c0b355de9b4c7932a4e41462192ec00e9a012f7871494ecb0f2916"} Aug 01 00:29:15.222843 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:15.222816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kmmsg" event={"ID":"2b480d6a-2902-469c-9b07-401b5826775a","Type":"ContainerStarted","Data":"1226da767bbbca7ead6ff2e220f86633f7b9e2ecf66a8649e952a887383b5942"} Aug 01 00:29:16.158194 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:16.157241 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:16.158194 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.157401 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:16.231000 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:16.230966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" event={"ID":"926883367cca128b6b84390dda10a55b","Type":"ContainerStarted","Data":"ef4ab0fe09c345478c898d811b974bee86ddcd83bfed2d8d728d4afa353c4ea4"} Aug 01 00:29:16.245071 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:16.245002 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-111.ec2.internal" podStartSLOduration=4.244981779 podStartE2EDuration="4.244981779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:15.245589849 +0000 UTC m=+4.602381509" watchObservedRunningTime="2026-08-01 00:29:16.244981779 +0000 UTC m=+5.601773436" Aug 01 00:29:16.669055 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:16.669018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:16.669249 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.669172 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:16.669249 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.669247 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret podName:6e26feb7-e8a1-43f0-a20d-a18db3aa016b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:20.66922665 +0000 UTC m=+10.026018309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret") pod "global-pull-secret-syncer-gkwsd" (UID: "6e26feb7-e8a1-43f0-a20d-a18db3aa016b") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:16.769493 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:16.769453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:16.769668 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.769627 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:16.769741 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.769700 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs podName:84fd6a55-cd43-4cfd-8750-41c249783dd8 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:20.769681956 +0000 UTC m=+10.126473596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs") pod "network-metrics-daemon-kkp8g" (UID: "84fd6a55-cd43-4cfd-8750-41c249783dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:16.871112 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:16.870471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:16.871112 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.870652 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:16.871112 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.870672 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:16.871112 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.870684 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jmnwl for pod openshift-network-diagnostics/network-check-target-p9lqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:16.871112 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:16.870754 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl podName:234cbdba-5336-48c0-9814-fcc0cdad9272 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:20.870732678 +0000 UTC m=+10.227524335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmnwl" (UniqueName: "kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl") pod "network-check-target-p9lqk" (UID: "234cbdba-5336-48c0-9814-fcc0cdad9272") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:17.158249 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:17.158120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:17.158722 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:17.158275 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:17.158793 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:17.158722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:17.158845 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:17.158816 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:18.157942 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:18.157903 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:18.158143 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:18.158068 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:19.158120 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:19.158083 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:19.158596 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:19.158255 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:19.158596 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:19.158569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:19.158706 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:19.158660 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:20.157336 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:20.157298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:20.157531 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.157437 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:20.710058 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:20.709429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:20.710058 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.709624 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:20.710058 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.709692 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret podName:6e26feb7-e8a1-43f0-a20d-a18db3aa016b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:28.709673415 +0000 UTC m=+18.066465053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret") pod "global-pull-secret-syncer-gkwsd" (UID: "6e26feb7-e8a1-43f0-a20d-a18db3aa016b") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:20.810001 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:20.809954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:20.810243 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.810197 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:20.810317 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.810265 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs podName:84fd6a55-cd43-4cfd-8750-41c249783dd8 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:28.810245332 +0000 UTC m=+18.167036972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs") pod "network-metrics-daemon-kkp8g" (UID: "84fd6a55-cd43-4cfd-8750-41c249783dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:20.911738 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:20.911000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:20.911738 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.911197 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:20.911738 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.911258 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:20.911738 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.911288 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jmnwl for pod openshift-network-diagnostics/network-check-target-p9lqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:20.911738 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:20.911382 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl podName:234cbdba-5336-48c0-9814-fcc0cdad9272 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:28.911361947 +0000 UTC m=+18.268153582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmnwl" (UniqueName: "kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl") pod "network-check-target-p9lqk" (UID: "234cbdba-5336-48c0-9814-fcc0cdad9272") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:21.159582 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:21.158909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:21.159582 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:21.159048 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:21.159966 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:21.159941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:21.160091 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:21.160047 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:22.157599 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:22.157566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:22.158069 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:22.157723 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:23.157485 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:23.157453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:23.157663 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:23.157590 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:23.158063 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:23.158044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:23.158203 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:23.158178 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:24.157861 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:24.157825 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:24.158377 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:24.157955 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:25.157332 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:25.157294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:25.157513 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:25.157435 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:25.157513 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:25.157483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:25.157637 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:25.157573 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:26.157936 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:26.157897 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:26.158398 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:26.158040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:27.157937 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:27.157893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:27.158460 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:27.158040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:27.158460 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:27.158113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:27.158460 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:27.158253 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:28.157152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:28.157110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:28.157359 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.157282 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:28.764679 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:28.764641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:28.765105 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.764784 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:28.765105 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.764850 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret podName:6e26feb7-e8a1-43f0-a20d-a18db3aa016b nodeName:}" failed. No retries permitted until 2026-08-01 00:29:44.764831446 +0000 UTC m=+34.121623088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret") pod "global-pull-secret-syncer-gkwsd" (UID: "6e26feb7-e8a1-43f0-a20d-a18db3aa016b") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:28.865385 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:28.865342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:28.865582 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.865514 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:28.865658 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.865592 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs podName:84fd6a55-cd43-4cfd-8750-41c249783dd8 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:44.865569375 +0000 UTC m=+34.222361011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs") pod "network-metrics-daemon-kkp8g" (UID: "84fd6a55-cd43-4cfd-8750-41c249783dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:28.966188 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:28.966096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:28.966340 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.966286 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:28.966340 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.966315 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:28.966340 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.966327 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jmnwl for pod openshift-network-diagnostics/network-check-target-p9lqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:28.966507 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:28.966394 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl podName:234cbdba-5336-48c0-9814-fcc0cdad9272 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:44.966373157 +0000 UTC m=+34.323164810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmnwl" (UniqueName: "kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl") pod "network-check-target-p9lqk" (UID: "234cbdba-5336-48c0-9814-fcc0cdad9272") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:29.157094 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:29.157059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:29.157283 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:29.157209 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:29.157283 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:29.157243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:29.157393 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:29.157330 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:30.158018 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:30.157986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:30.158485 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:30.158115 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:31.158938 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:31.158898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:31.159406 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:31.159022 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:31.159406 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:31.159122 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:31.159406 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:31.159174 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:32.157839 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:32.157800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:32.158030 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:32.157942 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:33.157805 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.157545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:33.158385 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.157545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:33.158385 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:33.157928 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:33.158385 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:33.158006 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:33.272865 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.272698 2575 generic.go:356] "Generic (PLEG): container finished" podID="171e54b7-3ce9-4b0d-96cc-477bdac0a271" containerID="f93d384682d69e1e9d2a1c6180e64f1c94f294f373be21f80ff253d6aeab226a" exitCode=0 Aug 01 00:29:33.273004 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.272761 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerDied","Data":"f93d384682d69e1e9d2a1c6180e64f1c94f294f373be21f80ff253d6aeab226a"} Aug 01 00:29:33.274493 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.274473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tv49p" event={"ID":"131e1d80-8e9d-4665-82f5-b0e3eefeca8c","Type":"ContainerStarted","Data":"8015e77044d41d78c40cb438d2b46e20bfb7d4e1109aa924f60e3797de4f2852"} Aug 01 00:29:33.275922 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.275836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l7qqq" event={"ID":"5e315e32-14a6-4ca3-8015-3593e3d3026c","Type":"ContainerStarted","Data":"d4c75aa26a57a70117bb043731ab7f5693a392733bf9502ce66640b06a29b77e"} Aug 01 00:29:33.279567 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.279542 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xpfmg" event={"ID":"7aa2e9a2-970b-41e5-a04a-9b5b75b5b499","Type":"ContainerStarted","Data":"ae077f6d07436c497075872bf83510d427d945c7c6108665e7e5cb2e175fdf11"} Aug 01 00:29:33.280816 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.280794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8h82v" event={"ID":"a16738b4-f53f-4150-a8e1-1f14a0abb0b1","Type":"ContainerStarted","Data":"2a450d7e2da2a0ba0705b2078d568df1804726e6207ed9e32123f24ac59b6e52"} Aug 01 00:29:33.282055 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.282035 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zn797" event={"ID":"c03487da-17ad-4def-9420-a85d74d06f0b","Type":"ContainerStarted","Data":"4c9e665395bae5da61f61ebd63cf4d0ac387a964b450c3652ac3ed95d1dfaaf4"} Aug 01 00:29:33.283276 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.283254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" event={"ID":"0e31ca4c-eb13-480c-83dd-f8d1988c736a","Type":"ContainerStarted","Data":"d9ebdb83bb84ff86dc10174dc89e6a65dddd38332fcf4bbb955b05a4ead5fe43"} Aug 01 00:29:33.284908 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.284892 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:29:33.285214 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.285196 2575 generic.go:356] "Generic (PLEG): container finished" podID="bf6cd5d5-6fee-4e49-bda7-30a29a317638" containerID="9351a142e34bf6d3eeada51bb5ab556d2bc2587debfdbfb5d9c0db69fa7e0e02" exitCode=1 Aug 01 00:29:33.285288 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.285223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"108611fc6efa298d431ad151de66b8f2735ba8d52c5cb39bf084f108c927a735"} Aug 01 00:29:33.285288 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.285267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerDied","Data":"9351a142e34bf6d3eeada51bb5ab556d2bc2587debfdbfb5d9c0db69fa7e0e02"} Aug 01 00:29:33.285351 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.285299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"0ba5e0049d0e092b6456ccf00478335f3ab43b44d1f832bd58d356cce4281dba"} Aug 01 00:29:33.297100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.297060 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-111.ec2.internal" podStartSLOduration=21.297045217 podStartE2EDuration="21.297045217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:16.245873833 +0000 UTC m=+5.602665490" watchObservedRunningTime="2026-08-01 00:29:33.297045217 +0000 UTC m=+22.653836874" Aug 01 00:29:33.310433 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.310388 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l7qqq" podStartSLOduration=3.972923364 podStartE2EDuration="22.310371764s" totalImagesPullingTime="18.3374484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:33.310054373 +0000 UTC m=+22.666846028" watchObservedRunningTime="2026-08-01 00:29:33.310371764 +0000 UTC m=+22.667163420" Aug 01 00:29:33.340644 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.340582 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8h82v" podStartSLOduration=3.999885373 podStartE2EDuration="22.340560018s" totalImagesPullingTime="18.340674645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:33.326465661 +0000 UTC m=+22.683257318" watchObservedRunningTime="2026-08-01 00:29:33.340560018 +0000 UTC m=+22.697351678" Aug 01 00:29:33.340774 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.340745 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xpfmg" podStartSLOduration=4.207073847 podStartE2EDuration="22.340738177s" totalImagesPullingTime="18.13366433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:33.34033167 +0000 UTC m=+22.697123316" watchObservedRunningTime="2026-08-01 00:29:33.340738177 +0000 UTC m=+22.697529834" Aug 01 00:29:33.358674 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.358622 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zn797" podStartSLOduration=3.972215249 podStartE2EDuration="22.358606153s" totalImagesPullingTime="18.386390904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:33.358220236 +0000 UTC m=+22.715011891" watchObservedRunningTime="2026-08-01 00:29:33.358606153 +0000 UTC m=+22.715397808" Aug 01 00:29:33.372217 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.372175 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tv49p" podStartSLOduration=8.123628971 podStartE2EDuration="22.372141693s" totalImagesPullingTime="14.248512722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:33.371920042 +0000 UTC m=+22.728711700" watchObservedRunningTime="2026-08-01 00:29:33.372141693 +0000 UTC m=+22.728933348" Aug 01 00:29:33.504400 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.504345 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:33.561570 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:33.561507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:34.158041 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.158017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:34.158533 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:34.158126 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:34.261880 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.261855 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 01 00:29:34.290317 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.290283 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kmmsg" event={"ID":"2b480d6a-2902-469c-9b07-401b5826775a","Type":"ContainerStarted","Data":"c04497ecff15cffa0e44c6b54c6d1fa924c026a8d90bb81985df042d5a3f4670"} Aug 01 00:29:34.291828 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.291805 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" event={"ID":"0e31ca4c-eb13-480c-83dd-f8d1988c736a","Type":"ContainerStarted","Data":"cac1c7e267657ec77092124f764a9b251a1b71a64cff0f2532bee9da361143b5"} Aug 01 00:29:34.294385 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.294334 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:29:34.294697 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.294675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"eb540a3870591eec4ebf4dc3237531089d2ca2834b90979adf2f01a05cc91d8d"} Aug 01 00:29:34.294761 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.294743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"c1e6e165a1eb072bfa31f8013cbc1c8e840ef021b13715203f01908fccec632f"} Aug 01 00:29:34.294803 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.294773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"3edea535340826f32851f5bdcf5fb53ff8a743cbfd763104559ea238c7dd821c"} Aug 01 00:29:34.300820 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.300794 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:34.304720 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.304681 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kmmsg" podStartSLOduration=5.171691704 podStartE2EDuration="23.304668767s" totalImagesPullingTime="18.132977063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:34.304370044 +0000 UTC m=+23.661161699" watchObservedRunningTime="2026-08-01 00:29:34.304668767 +0000 UTC m=+23.661460423" Aug 01 00:29:34.403141 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.403100 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:34.460274 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:34.460238 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:35.084910 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.084778 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-01T00:29:34.261872646Z","UUID":"9ad0762d-edf4-4501-bc7a-e27ef11eb369","Handler":null,"Name":"","Endpoint":""} Aug 01 00:29:35.086489 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.086460 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 01 00:29:35.086606 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.086509 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 01 00:29:35.157568 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.157520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:35.157568 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.157572 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:35.157818 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:35.157703 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:35.157980 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:35.157919 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:35.191613 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.191570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:35.592077 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.592029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:35.693168 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:35.693066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:36.157538 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:36.157511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:36.157725 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:36.157645 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:36.303206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:36.302908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" event={"ID":"0e31ca4c-eb13-480c-83dd-f8d1988c736a","Type":"ContainerStarted","Data":"82e9a4b6c52091b46aeaa2892f463a95363139766c8de51298b738cdc6fd965d"} Aug 01 00:29:36.306265 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:36.306239 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:29:36.306737 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:36.306711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"ab2f8224f9d9856ed7a87297c6dd88c5d7aaa79dc7bd60d79a9876a7eb01841f"} Aug 01 00:29:36.321010 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:36.320955 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhtq8" podStartSLOduration=4.930848468 podStartE2EDuration="25.320934648s" totalImagesPullingTime="20.39008618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:36.320627874 +0000 UTC m=+25.677419530" watchObservedRunningTime="2026-08-01 00:29:36.320934648 +0000 UTC m=+25.677726305" Aug 01 00:29:37.157911 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:37.157877 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:37.158137 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:37.157881 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:37.158137 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:37.158027 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:37.158137 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:37.158096 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:37.661018 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:37.660976 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:37.661673 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:37.661649 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:38.157219 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.157030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:38.157381 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:38.157305 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:38.313261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.313227 2575 generic.go:356] "Generic (PLEG): container finished" podID="171e54b7-3ce9-4b0d-96cc-477bdac0a271" containerID="d5bdf6b6a739e6d71083ff348f640a360573dfe0342e1fa08f1d73b19a663ed8" exitCode=0 Aug 01 00:29:38.313431 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.313306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerDied","Data":"d5bdf6b6a739e6d71083ff348f640a360573dfe0342e1fa08f1d73b19a663ed8"} Aug 01 00:29:38.316897 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.316875 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:29:38.317302 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.317279 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"711e668fc9d6c829011e6596bd1d0fe489c96c5992ff82e6547b90b86d91589c"} Aug 01 00:29:38.317458 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.317446 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:38.317737 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.317724 2575 scope.go:117] "RemoveContainer" containerID="9351a142e34bf6d3eeada51bb5ab556d2bc2587debfdbfb5d9c0db69fa7e0e02" Aug 01 00:29:38.317993 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.317977 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l7qqq" Aug 01 00:29:38.438609 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.438488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:38.807994 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.807899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:38.907984 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:38.907952 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:39.157725 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.157436 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:39.157725 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:39.157561 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:39.157725 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.157692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:39.157952 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:39.157799 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:39.322009 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.321977 2575 generic.go:356] "Generic (PLEG): container finished" podID="171e54b7-3ce9-4b0d-96cc-477bdac0a271" containerID="508e6ca350e0973bd48e97e0f8fdf57a758a3c82a4d4013c1d1036b679fbbc11" exitCode=0 Aug 01 00:29:39.322207 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.322023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerDied","Data":"508e6ca350e0973bd48e97e0f8fdf57a758a3c82a4d4013c1d1036b679fbbc11"} Aug 01 00:29:39.326487 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.326465 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:29:39.326882 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.326850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" event={"ID":"bf6cd5d5-6fee-4e49-bda7-30a29a317638","Type":"ContainerStarted","Data":"28ca4421bfbd037338b3ed36fc1c16dc74a6b4d53fbfe32f447d5fb1b4d15660"} Aug 01 00:29:39.327259 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.327239 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:39.327359 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.327311 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:39.327420 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.327364 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:39.342962 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.342887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:39.343084 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.343040 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:29:39.367424 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.367364 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" podStartSLOduration=10.020878949 podStartE2EDuration="28.367348199s" totalImagesPullingTime="18.34646925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:39.365789847 +0000 UTC m=+28.722581504" watchObservedRunningTime="2026-08-01 00:29:39.367348199 +0000 UTC m=+28.724139855" Aug 01 00:29:39.853468 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.853435 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p9lqk"] Aug 01 00:29:39.853870 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.853556 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:39.853870 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:39.853673 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:39.856031 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.856003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kkp8g"] Aug 01 00:29:39.856212 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.856136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:39.856296 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:39.856273 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:39.856912 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.856882 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gkwsd"] Aug 01 00:29:39.857026 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:39.856978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:39.857090 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:39.857068 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:40.331754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:40.331506 2575 generic.go:356] "Generic (PLEG): container finished" podID="171e54b7-3ce9-4b0d-96cc-477bdac0a271" containerID="4d7374f2c3f394d660fe8c150f7ae769d398dfd422e20547ac16e8f3f474db8b" exitCode=0 Aug 01 00:29:40.331754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:40.331588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerDied","Data":"4d7374f2c3f394d660fe8c150f7ae769d398dfd422e20547ac16e8f3f474db8b"} Aug 01 00:29:41.158448 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:41.158412 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:41.158977 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:41.158488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:41.158977 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:41.158502 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:41.158977 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:41.158622 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:41.158977 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:41.158637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:41.158977 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:41.158712 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:43.158042 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:43.158003 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:43.158898 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:43.158005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:43.158898 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:43.158152 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:43.158898 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:43.158013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:43.158898 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:43.158217 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:43.158898 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:43.158312 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:44.299825 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:44.299784 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:44.399788 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:44.399686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:44.778377 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:44.778341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:44.778543 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.778458 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:44.778543 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.778517 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret podName:6e26feb7-e8a1-43f0-a20d-a18db3aa016b nodeName:}" failed. No retries permitted until 2026-08-01 00:30:16.778500877 +0000 UTC m=+66.135292518 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret") pod "global-pull-secret-syncer-gkwsd" (UID: "6e26feb7-e8a1-43f0-a20d-a18db3aa016b") : object "kube-system"/"original-pull-secret" not registered Aug 01 00:29:44.878915 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:44.878883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:44.879083 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.879040 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:44.879128 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.879105 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs podName:84fd6a55-cd43-4cfd-8750-41c249783dd8 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:16.879088251 +0000 UTC m=+66.235879885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs") pod "network-metrics-daemon-kkp8g" (UID: "84fd6a55-cd43-4cfd-8750-41c249783dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 01 00:29:44.980279 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:44.980235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:44.980449 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.980365 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 01 00:29:44.980449 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.980383 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 01 00:29:44.980449 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.980392 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jmnwl for pod openshift-network-diagnostics/network-check-target-p9lqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:44.980449 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:44.980444 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl podName:234cbdba-5336-48c0-9814-fcc0cdad9272 nodeName:}" failed. No retries permitted until 2026-08-01 00:30:16.980428287 +0000 UTC m=+66.337219923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmnwl" (UniqueName: "kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl") pod "network-check-target-p9lqk" (UID: "234cbdba-5336-48c0-9814-fcc0cdad9272") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 01 00:29:45.157109 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.157079 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:45.157109 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.157098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:45.157326 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.157079 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:45.157326 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:45.157207 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkwsd" podUID="6e26feb7-e8a1-43f0-a20d-a18db3aa016b" Aug 01 00:29:45.157408 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:45.157321 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p9lqk" podUID="234cbdba-5336-48c0-9814-fcc0cdad9272" Aug 01 00:29:45.157442 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:45.157407 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkp8g" podUID="84fd6a55-cd43-4cfd-8750-41c249783dd8" Aug 01 00:29:45.281215 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.281182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:45.338575 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.338535 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:45.348214 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.348186 2575 generic.go:356] "Generic (PLEG): container finished" podID="171e54b7-3ce9-4b0d-96cc-477bdac0a271" containerID="e80620a48445eb0529def18516e1f4670b238b01296af4e72e404bc0c55ebb51" exitCode=0 Aug 01 00:29:45.348372 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.348237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerDied","Data":"e80620a48445eb0529def18516e1f4670b238b01296af4e72e404bc0c55ebb51"} Aug 01 00:29:45.460516 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.460490 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-111.ec2.internal" event="NodeReady" Aug 01 00:29:45.460671 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.460640 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 01 00:29:45.505874 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.505844 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rbqbf"] Aug 01 00:29:45.521510 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.521480 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-48ftq"] Aug 01 00:29:45.521634 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.521610 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.524110 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.524049 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 01 00:29:45.524110 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.524076 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 01 00:29:45.524389 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.524373 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fl89b\"" Aug 01 00:29:45.533065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.533041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rbqbf"] Aug 01 00:29:45.533065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.533076 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48ftq"] Aug 01 00:29:45.533254 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.533148 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.535687 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.535669 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 01 00:29:45.536025 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.536004 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 01 00:29:45.536096 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.536004 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-stmw4\"" Aug 01 00:29:45.536096 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.536009 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 01 00:29:45.584944 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.584923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-config-volume\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.585062 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.584954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.585062 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.585052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-tmp-dir\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.585149 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.585092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94t8\" (UniqueName: \"kubernetes.io/projected/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-kube-api-access-t94t8\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.617710 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.617613 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:45.685546 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.685509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.685729 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.685593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-tmp-dir\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.685729 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.685647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94t8\" (UniqueName: \"kubernetes.io/projected/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-kube-api-access-t94t8\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.685729 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:45.685668 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:45.685879 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:45.685732 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls podName:afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a nodeName:}" failed. No retries permitted until 2026-08-01 00:29:46.185715793 +0000 UTC m=+35.542507428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls") pod "dns-default-rbqbf" (UID: "afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a") : secret "dns-default-metrics-tls" not found Aug 01 00:29:45.685879 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.685768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47w77\" (UniqueName: \"kubernetes.io/projected/e1cdfc70-dd52-486f-a9df-2edf6cd2b626-kube-api-access-47w77\") pod \"ingress-canary-48ftq\" (UID: \"e1cdfc70-dd52-486f-a9df-2edf6cd2b626\") " pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.685879 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.685807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1cdfc70-dd52-486f-a9df-2edf6cd2b626-cert\") pod \"ingress-canary-48ftq\" (UID: \"e1cdfc70-dd52-486f-a9df-2edf6cd2b626\") " pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.685879 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.685870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-config-volume\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.686028 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.685972 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-tmp-dir\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.686361 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.686345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-config-volume\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.702704 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.702667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94t8\" (UniqueName: \"kubernetes.io/projected/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-kube-api-access-t94t8\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:45.717845 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.717815 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:45.786545 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.786278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47w77\" (UniqueName: \"kubernetes.io/projected/e1cdfc70-dd52-486f-a9df-2edf6cd2b626-kube-api-access-47w77\") pod \"ingress-canary-48ftq\" (UID: \"e1cdfc70-dd52-486f-a9df-2edf6cd2b626\") " pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.786713 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.786560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1cdfc70-dd52-486f-a9df-2edf6cd2b626-cert\") pod \"ingress-canary-48ftq\" (UID: \"e1cdfc70-dd52-486f-a9df-2edf6cd2b626\") " pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.788994 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.788976 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1cdfc70-dd52-486f-a9df-2edf6cd2b626-cert\") pod \"ingress-canary-48ftq\" (UID: \"e1cdfc70-dd52-486f-a9df-2edf6cd2b626\") " pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.795766 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.795712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47w77\" (UniqueName: \"kubernetes.io/projected/e1cdfc70-dd52-486f-a9df-2edf6cd2b626-kube-api-access-47w77\") pod \"ingress-canary-48ftq\" (UID: \"e1cdfc70-dd52-486f-a9df-2edf6cd2b626\") " pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.808986 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.808958 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k"] Aug 01 00:29:45.813987 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.813951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:45.832765 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.832728 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k"] Aug 01 00:29:45.832931 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.832859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" Aug 01 00:29:45.835493 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.835469 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 01 00:29:45.835623 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.835499 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ptnj5\"" Aug 01 00:29:45.835754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.835735 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 01 00:29:45.866583 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.866551 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v67t2"] Aug 01 00:29:45.887043 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.886951 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48ftq" Aug 01 00:29:45.894364 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.894334 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v67t2"] Aug 01 00:29:45.894498 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.894452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:45.896664 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.896642 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 01 00:29:45.896914 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.896897 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 01 00:29:45.896983 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.896924 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9ldw2\"" Aug 01 00:29:45.897141 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.897124 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 01 00:29:45.897276 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.897181 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 01 00:29:45.903719 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.903415 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:45.980383 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.980344 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:45.988216 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.988185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptdj\" (UniqueName: \"kubernetes.io/projected/662095f4-3846-4be3-b1c6-3b29fd533d83-kube-api-access-lptdj\") pod \"migrator-68bc745d87-wrj2k\" (UID: \"662095f4-3846-4be3-b1c6-3b29fd533d83\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" Aug 01 00:29:45.988387 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.988231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e119374b-fa39-4b71-8856-d50e943d4c4c-data-volume\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:45.988387 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.988260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:45.988387 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.988335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e119374b-fa39-4b71-8856-d50e943d4c4c-crio-socket\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:45.988387 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.988367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvc8\" (UniqueName: \"kubernetes.io/projected/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-api-access-4pvc8\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:45.988538 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:45.988392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.085734 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.085679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48ftq"] Aug 01 00:29:46.085927 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.085882 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:46.089763 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.089738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptdj\" (UniqueName: \"kubernetes.io/projected/662095f4-3846-4be3-b1c6-3b29fd533d83-kube-api-access-lptdj\") pod \"migrator-68bc745d87-wrj2k\" (UID: \"662095f4-3846-4be3-b1c6-3b29fd533d83\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" Aug 01 00:29:46.089872 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.089779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e119374b-fa39-4b71-8856-d50e943d4c4c-data-volume\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.089872 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.089805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.089872 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.089841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e119374b-fa39-4b71-8856-d50e943d4c4c-crio-socket\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.089872 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.089864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvc8\" (UniqueName: \"kubernetes.io/projected/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-api-access-4pvc8\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.090074 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.089888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.090074 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.089905 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:46.090074 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.089998 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:46.589975949 +0000 UTC m=+35.946767604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : configmap "kube-rbac-proxy" not found Aug 01 00:29:46.090074 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.090003 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:46.090074 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.090058 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:46.590042629 +0000 UTC m=+35.946834263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:46.090355 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.090137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e119374b-fa39-4b71-8856-d50e943d4c4c-crio-socket\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.090355 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.090237 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e119374b-fa39-4b71-8856-d50e943d4c4c-data-volume\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.090945 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:46.090923 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cdfc70_dd52_486f_a9df_2edf6cd2b626.slice/crio-d4363b3c169f27bf575d88926d80a84b5577390ef2d6124e7f3ee22fa45c9b9f WatchSource:0}: Error finding container d4363b3c169f27bf575d88926d80a84b5577390ef2d6124e7f3ee22fa45c9b9f: Status 404 returned error can't find the container with id d4363b3c169f27bf575d88926d80a84b5577390ef2d6124e7f3ee22fa45c9b9f Aug 01 00:29:46.099793 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.099759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvc8\" (UniqueName: \"kubernetes.io/projected/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-api-access-4pvc8\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.099907 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.099759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptdj\" (UniqueName: \"kubernetes.io/projected/662095f4-3846-4be3-b1c6-3b29fd533d83-kube-api-access-lptdj\") pod \"migrator-68bc745d87-wrj2k\" (UID: \"662095f4-3846-4be3-b1c6-3b29fd533d83\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" Aug 01 00:29:46.142261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.142137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" Aug 01 00:29:46.190675 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.190641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:46.190843 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.190797 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:46.190882 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.190867 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls podName:afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a nodeName:}" failed. No retries permitted until 2026-08-01 00:29:47.190849449 +0000 UTC m=+36.547641083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls") pod "dns-default-rbqbf" (UID: "afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a") : secret "dns-default-metrics-tls" not found Aug 01 00:29:46.272241 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.272207 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k"] Aug 01 00:29:46.275969 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:46.275944 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662095f4_3846_4be3_b1c6_3b29fd533d83.slice/crio-52f8fa141dd3c6fdb26fc7cffc4098d6807f9f39dd7258c690f72b04f599e33d WatchSource:0}: Error finding container 52f8fa141dd3c6fdb26fc7cffc4098d6807f9f39dd7258c690f72b04f599e33d: Status 404 returned error can't find the container with id 52f8fa141dd3c6fdb26fc7cffc4098d6807f9f39dd7258c690f72b04f599e33d Aug 01 00:29:46.353651 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.353611 2575 generic.go:356] "Generic (PLEG): container finished" podID="171e54b7-3ce9-4b0d-96cc-477bdac0a271" containerID="0611b2e7240cb4d45c4dcd7f0ed8b3ce2d5b1eee3f9b06d4e12d0772bfc4c421" exitCode=0 Aug 01 00:29:46.354214 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.353698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerDied","Data":"0611b2e7240cb4d45c4dcd7f0ed8b3ce2d5b1eee3f9b06d4e12d0772bfc4c421"} Aug 01 00:29:46.354926 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.354893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48ftq" event={"ID":"e1cdfc70-dd52-486f-a9df-2edf6cd2b626","Type":"ContainerStarted","Data":"d4363b3c169f27bf575d88926d80a84b5577390ef2d6124e7f3ee22fa45c9b9f"} Aug 01 00:29:46.355848 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.355831 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" event={"ID":"662095f4-3846-4be3-b1c6-3b29fd533d83","Type":"ContainerStarted","Data":"52f8fa141dd3c6fdb26fc7cffc4098d6807f9f39dd7258c690f72b04f599e33d"} Aug 01 00:29:46.594669 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.594478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.594846 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.594630 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:46.594846 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.594771 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:46.594846 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.594792 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:47.594776829 +0000 UTC m=+36.951568463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:46.594846 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:46.594728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:46.594846 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:46.594809 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:47.594798835 +0000 UTC m=+36.951590472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : configmap "kube-rbac-proxy" not found Aug 01 00:29:47.157875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.157845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:29:47.158861 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.158484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:29:47.158861 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.158514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:29:47.160883 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.160601 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnzmh\"" Aug 01 00:29:47.160883 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.160613 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 01 00:29:47.161754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.161630 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 01 00:29:47.161754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.161696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 01 00:29:47.161754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.161718 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 01 00:29:47.162013 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.161936 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2x5h\"" Aug 01 00:29:47.199595 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.199556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:47.200058 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:47.200018 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:47.200224 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:47.200129 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls podName:afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a nodeName:}" failed. No retries permitted until 2026-08-01 00:29:49.200084004 +0000 UTC m=+38.556875657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls") pod "dns-default-rbqbf" (UID: "afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a") : secret "dns-default-metrics-tls" not found Aug 01 00:29:47.363913 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.363803 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" event={"ID":"171e54b7-3ce9-4b0d-96cc-477bdac0a271","Type":"ContainerStarted","Data":"5e86c9ef8ffc058cfca0b60c839939ca86824799ca7f39af80264ab8fcc2505e"} Aug 01 00:29:47.385268 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.385208 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b7zfn" podStartSLOduration=7.771511105 podStartE2EDuration="36.385187195s" totalImagesPullingTime="28.61367609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:47.384143981 +0000 UTC m=+36.740935638" watchObservedRunningTime="2026-08-01 00:29:47.385187195 +0000 UTC m=+36.741978849" Aug 01 00:29:47.506864 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.506766 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:47.589317 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.589262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:47.592280 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.592253 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-d6xph"] Aug 01 00:29:47.603640 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.603610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:47.603810 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.603658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:47.603810 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:47.603759 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:47.603911 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:47.603819 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:47.603911 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:47.603836 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:49.603816117 +0000 UTC m=+38.960607755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : configmap "kube-rbac-proxy" not found Aug 01 00:29:47.603911 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:47.603872 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:49.603855383 +0000 UTC m=+38.960647022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:47.607842 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.607807 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-d6xph"] Aug 01 00:29:47.608019 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.607907 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.610543 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.610519 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 01 00:29:47.611246 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.611222 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rckdx\"" Aug 01 00:29:47.611375 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.611352 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 01 00:29:47.611466 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.611411 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 01 00:29:47.611541 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.611519 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 01 00:29:47.652261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.652213 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:47.704800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.704765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ba4ed77-dff5-4801-8213-6390f323796a-signing-key\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.704974 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.704841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrw7\" (UniqueName: \"kubernetes.io/projected/3ba4ed77-dff5-4801-8213-6390f323796a-kube-api-access-kdrw7\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.704974 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.704867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ba4ed77-dff5-4801-8213-6390f323796a-signing-cabundle\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.806060 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.805973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrw7\" (UniqueName: \"kubernetes.io/projected/3ba4ed77-dff5-4801-8213-6390f323796a-kube-api-access-kdrw7\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.806060 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.806022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ba4ed77-dff5-4801-8213-6390f323796a-signing-cabundle\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.806315 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.806085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ba4ed77-dff5-4801-8213-6390f323796a-signing-key\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.806798 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.806772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ba4ed77-dff5-4801-8213-6390f323796a-signing-cabundle\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.810305 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.810278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ba4ed77-dff5-4801-8213-6390f323796a-signing-key\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.814821 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.814798 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrw7\" (UniqueName: \"kubernetes.io/projected/3ba4ed77-dff5-4801-8213-6390f323796a-kube-api-access-kdrw7\") pod \"service-ca-98c88fbb5-d6xph\" (UID: \"3ba4ed77-dff5-4801-8213-6390f323796a\") " pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:47.918421 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:47.918378 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" Aug 01 00:29:48.175561 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:48.175506 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:48.682815 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:48.682780 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:48.831941 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:48.831913 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpfmg_7aa2e9a2-970b-41e5-a04a-9b5b75b5b499/dns-node-resolver/0.log" Aug 01 00:29:48.845496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:48.845460 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-d6xph"] Aug 01 00:29:48.850099 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:29:48.850062 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba4ed77_dff5_4801_8213_6390f323796a.slice/crio-333eed72685d5ab6da3a5af2c79845185dee83527017862c05fa41e6cbaec668 WatchSource:0}: Error finding container 333eed72685d5ab6da3a5af2c79845185dee83527017862c05fa41e6cbaec668: Status 404 returned error can't find the container with id 333eed72685d5ab6da3a5af2c79845185dee83527017862c05fa41e6cbaec668 Aug 01 00:29:49.219941 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.219754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:49.220125 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:49.219916 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:49.220125 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:49.220016 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls podName:afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a nodeName:}" failed. No retries permitted until 2026-08-01 00:29:53.220001319 +0000 UTC m=+42.576792953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls") pod "dns-default-rbqbf" (UID: "afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a") : secret "dns-default-metrics-tls" not found Aug 01 00:29:49.370746 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.370666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48ftq" event={"ID":"e1cdfc70-dd52-486f-a9df-2edf6cd2b626","Type":"ContainerStarted","Data":"280fb086a7a8bfc93ce55b522883e89ddf52fdc303ab3a5cc639084acca63812"} Aug 01 00:29:49.372697 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.372669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" event={"ID":"662095f4-3846-4be3-b1c6-3b29fd533d83","Type":"ContainerStarted","Data":"0055c9a07198a5345787bae57586c14618bfad1db8260a3a5e6ad513c04c4c53"} Aug 01 00:29:49.372855 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.372715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" event={"ID":"662095f4-3846-4be3-b1c6-3b29fd533d83","Type":"ContainerStarted","Data":"8de9df40bcae4e2b9e557d24ead00d53c399342e1898f9b5923ef6de134ceb5b"} Aug 01 00:29:49.373904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.373882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" event={"ID":"3ba4ed77-dff5-4801-8213-6390f323796a","Type":"ContainerStarted","Data":"333eed72685d5ab6da3a5af2c79845185dee83527017862c05fa41e6cbaec668"} Aug 01 00:29:49.388022 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.387972 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-48ftq" podStartSLOduration=1.45032917 podStartE2EDuration="4.38795364s" totalImagesPullingTime="2.93762447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:49.387192426 +0000 UTC m=+38.743984084" watchObservedRunningTime="2026-08-01 00:29:49.38795364 +0000 UTC m=+38.744745274" Aug 01 00:29:49.406551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.406499 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-wrj2k" podStartSLOduration=1.963664923 podStartE2EDuration="4.406481453s" totalImagesPullingTime="2.44281653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:49.406473459 +0000 UTC m=+38.763265106" watchObservedRunningTime="2026-08-01 00:29:49.406481453 +0000 UTC m=+38.763273108" Aug 01 00:29:49.623327 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.623232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:49.623327 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.623302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:49.623528 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:49.623379 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:49.623528 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:49.623430 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:49.623528 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:49.623454 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:53.623432413 +0000 UTC m=+42.980224055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : configmap "kube-rbac-proxy" not found Aug 01 00:29:49.623528 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:49.623477 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:29:53.62346325 +0000 UTC m=+42.980254897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:49.833346 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:49.833315 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tv49p_131e1d80-8e9d-4665-82f5-b0e3eefeca8c/node-ca/0.log" Aug 01 00:29:50.233235 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:50.233198 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-48ftq_e1cdfc70-dd52-486f-a9df-2edf6cd2b626/serve-healthcheck-canary/0.log" Aug 01 00:29:50.831301 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:50.831269 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-wrj2k_662095f4-3846-4be3-b1c6-3b29fd533d83/migrator/0.log" Aug 01 00:29:51.029045 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:51.029018 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-wrj2k_662095f4-3846-4be3-b1c6-3b29fd533d83/graceful-termination/0.log" Aug 01 00:29:51.276322 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:51.276288 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:51.377254 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:51.377217 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:51.380652 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:51.380626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" event={"ID":"3ba4ed77-dff5-4801-8213-6390f323796a","Type":"ContainerStarted","Data":"b163b6b984d5645305cf687f3cc3514e3cde803f337827bd45a9af6c68ab90c0"} Aug 01 00:29:51.397056 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:51.397005 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-d6xph" podStartSLOduration=2.022731945 podStartE2EDuration="4.396988207s" totalImagesPullingTime="2.374256262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:51.39628444 +0000 UTC m=+40.753076111" watchObservedRunningTime="2026-08-01 00:29:51.396988207 +0000 UTC m=+40.753779862" Aug 01 00:29:52.994115 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:52.994040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:53.120648 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.120612 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:53.251759 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.251674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:29:53.251929 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:53.251834 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 01 00:29:53.251929 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:53.251902 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls podName:afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a nodeName:}" failed. No retries permitted until 2026-08-01 00:30:01.251884177 +0000 UTC m=+50.608675825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls") pod "dns-default-rbqbf" (UID: "afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a") : secret "dns-default-metrics-tls" not found Aug 01 00:29:53.278852 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.278814 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:53.375133 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.375045 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:53.567115 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.567027 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:53.655400 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.655362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:53.655608 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:53.655505 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:29:53.655608 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.655516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:29:53.655608 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:53.655580 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:30:01.655558788 +0000 UTC m=+51.012350436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : secret "insights-runtime-extractor-tls" not found Aug 01 00:29:53.655971 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:53.655627 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 01 00:29:53.655971 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:29:53.655691 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:30:01.655673983 +0000 UTC m=+51.012465623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : configmap "kube-rbac-proxy" not found Aug 01 00:29:53.669979 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.669937 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:53.767717 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:53.767678 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:54.289858 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:54.289826 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:54.656951 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:54.656911 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:54.889498 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:54.889449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:55.310809 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:55.310763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:55.688076 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:55.688024 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:56.086253 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:56.086139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:56.282006 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:56.281965 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:56.484844 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:56.484802 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:56.696823 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:56.696790 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:56.986361 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:56.986327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:57.687842 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:57.687810 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:57.897673 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:57.897637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:57.952775 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:57.952680 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:58.085548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:58.085507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:58.284465 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:58.284378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:58.484952 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:58.484908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:58.948413 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:58.948384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:59.686514 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:59.686479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:29:59.892100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:29:59.892046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:00.087753 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:00.087668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:00.271406 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:00.271374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:00.318549 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:00.318514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:00.486261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:00.486225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:00.848933 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:00.848843 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:00.949611 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:00.949577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:01.061650 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.061611 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:01.162595 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.162558 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:01.325986 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.325947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:30:01.329421 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.329391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a-metrics-tls\") pod \"dns-default-rbqbf\" (UID: \"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a\") " pod="openshift-dns/dns-default-rbqbf" Aug 01 00:30:01.385911 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.385880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:01.482023 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.481919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rbqbf" Aug 01 00:30:01.504273 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.504235 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:01.623533 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.623502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rbqbf"] Aug 01 00:30:01.626943 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:01.626913 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa9e0f8_6c96_477e_b7ba_f0d1c0ba868a.slice/crio-93fad16d287b1b9c46d16dc2c89c9493da7e68556b5d486e841b8870e73400a8 WatchSource:0}: Error finding container 93fad16d287b1b9c46d16dc2c89c9493da7e68556b5d486e841b8870e73400a8: Status 404 returned error can't find the container with id 93fad16d287b1b9c46d16dc2c89c9493da7e68556b5d486e841b8870e73400a8 Aug 01 00:30:01.636820 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.636792 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:01.700294 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.700260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:01.729093 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.729060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:30:01.729261 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.729106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:30:01.729302 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:30:01.729266 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 01 00:30:01.729336 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:30:01.729324 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls podName:e119374b-fa39-4b71-8856-d50e943d4c4c nodeName:}" failed. No retries permitted until 2026-08-01 00:30:17.72930805 +0000 UTC m=+67.086099685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v67t2" (UID: "e119374b-fa39-4b71-8856-d50e943d4c4c") : secret "insights-runtime-extractor-tls" not found Aug 01 00:30:01.729848 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.729801 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e119374b-fa39-4b71-8856-d50e943d4c4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:30:01.945540 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:01.945510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:02.060431 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:02.060395 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:02.160983 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:02.160950 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:02.397860 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:02.397771 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:02.410336 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:02.410301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rbqbf" event={"ID":"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a","Type":"ContainerStarted","Data":"93fad16d287b1b9c46d16dc2c89c9493da7e68556b5d486e841b8870e73400a8"} Aug 01 00:30:02.529413 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:02.529369 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:02.684696 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:02.684658 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:02.785910 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:02.785876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:03.322958 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:03.322912 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:03.689778 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:03.689742 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:03.787544 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:03.787509 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:03.887250 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:03.887213 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:04.087802 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:04.087715 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:04.420344 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:04.420268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rbqbf" event={"ID":"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a","Type":"ContainerStarted","Data":"a9c4edce6678d6d50bdd22e0bafe5b61fc43271bec6d19f90c7acb70d1d9c33e"} Aug 01 00:30:05.425292 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:05.425259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rbqbf" event={"ID":"afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a","Type":"ContainerStarted","Data":"7244949b7e56a49dd6e058a30b52a75e7d13eb13fbd50239802e13b35c1c3fb1"} Aug 01 00:30:05.425678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:05.425477 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rbqbf" Aug 01 00:30:05.442021 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:05.441971 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rbqbf" podStartSLOduration=17.761796261 podStartE2EDuration="20.441953106s" totalImagesPullingTime="2.680156845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:05.441282296 +0000 UTC m=+54.798073951" watchObservedRunningTime="2026-08-01 00:30:05.441953106 +0000 UTC m=+54.798744765" Aug 01 00:30:06.415474 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:06.415437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:06.511922 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:06.511884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:08.560601 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.560566 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m"] Aug 01 00:30:08.562619 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.562597 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr"] Aug 01 00:30:08.562743 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.562708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.564943 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.564922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.565331 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.565312 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 01 00:30:08.565610 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.565589 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 01 00:30:08.565660 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.565619 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 01 00:30:08.565716 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.565679 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 01 00:30:08.567570 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.567551 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 01 00:30:08.567659 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.567555 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 01 00:30:08.567838 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.567822 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 01 00:30:08.567933 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.567913 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 01 00:30:08.574989 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.574962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m"] Aug 01 00:30:08.576149 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.576125 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr"] Aug 01 00:30:08.577674 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.577652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-ca\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.577769 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.577690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.577817 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.577781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-tmp\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.577859 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.577826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.577905 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.577853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsszl\" (UniqueName: \"kubernetes.io/projected/bfec6fe0-0408-4337-ae20-3424c0459db1-kube-api-access-tsszl\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.577962 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.577943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.578005 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.577977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-hub\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.578109 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.578089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec6fe0-0408-4337-ae20-3424c0459db1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.578150 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.578140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddzm9\" (UniqueName: \"kubernetes.io/projected/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-kube-api-access-ddzm9\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.671273 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.671235 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:08.679265 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-ca\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.679265 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.679515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-tmp\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.679515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.679515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsszl\" (UniqueName: \"kubernetes.io/projected/bfec6fe0-0408-4337-ae20-3424c0459db1-kube-api-access-tsszl\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.679515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.679515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-hub\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.679765 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec6fe0-0408-4337-ae20-3424c0459db1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.679765 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddzm9\" (UniqueName: \"kubernetes.io/projected/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-kube-api-access-ddzm9\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.679765 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.679750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-tmp\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.680517 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.680484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec6fe0-0408-4337-ae20-3424c0459db1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.681909 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.681881 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-ca\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.682143 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.682121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.682210 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.682168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.682210 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.682127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-hub\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.682460 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.682438 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bfec6fe0-0408-4337-ae20-3424c0459db1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.688279 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.688253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsszl\" (UniqueName: \"kubernetes.io/projected/bfec6fe0-0408-4337-ae20-3424c0459db1-kube-api-access-tsszl\") pod \"cluster-proxy-proxy-agent-7dd74686-xqftr\" (UID: \"bfec6fe0-0408-4337-ae20-3424c0459db1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:08.688553 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.688523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddzm9\" (UniqueName: \"kubernetes.io/projected/8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79-kube-api-access-ddzm9\") pod \"klusterlet-addon-workmgr-7bc45c5754-wdx4m\" (UID: \"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.747671 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.747633 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:08.846069 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.845970 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:08.873956 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.873918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:08.893027 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:08.893000 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" Aug 01 00:30:09.009337 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:09.009307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m"] Aug 01 00:30:09.012642 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:09.012609 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a366acf_ae5b_4ed8_8fd6_5c7f892f6f79.slice/crio-d64845b6122063cf399b15e2c0b5bddc44952e6bc4a8fc009d872a5cb5f7bf10 WatchSource:0}: Error finding container d64845b6122063cf399b15e2c0b5bddc44952e6bc4a8fc009d872a5cb5f7bf10: Status 404 returned error can't find the container with id d64845b6122063cf399b15e2c0b5bddc44952e6bc4a8fc009d872a5cb5f7bf10 Aug 01 00:30:09.026256 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:09.026224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr"] Aug 01 00:30:09.030303 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:09.030273 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfec6fe0_0408_4337_ae20_3424c0459db1.slice/crio-b33d05434238655f49b2fcb0a1152e7c5395ce49eb81815af314a3a553615f69 WatchSource:0}: Error finding container b33d05434238655f49b2fcb0a1152e7c5395ce49eb81815af314a3a553615f69: Status 404 returned error can't find the container with id b33d05434238655f49b2fcb0a1152e7c5395ce49eb81815af314a3a553615f69 Aug 01 00:30:09.438074 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:09.438040 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" event={"ID":"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79","Type":"ContainerStarted","Data":"d64845b6122063cf399b15e2c0b5bddc44952e6bc4a8fc009d872a5cb5f7bf10"} Aug 01 00:30:09.439014 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:09.438982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" event={"ID":"bfec6fe0-0408-4337-ae20-3424c0459db1","Type":"ContainerStarted","Data":"b33d05434238655f49b2fcb0a1152e7c5395ce49eb81815af314a3a553615f69"} Aug 01 00:30:11.349085 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:11.349051 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rt596" Aug 01 00:30:11.360034 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:11.359946 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:11.425698 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:11.425652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:11.484405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:11.484349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:11.532116 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:11.532078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:11.580869 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:11.580806 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:12.433367 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:12.433334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:12.539672 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:12.539637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:14.695147 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:14.695101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:14.796544 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:14.796333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:14.850732 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:14.850692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:15.048793 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:15.048712 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:15.431213 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:15.431184 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rbqbf" Aug 01 00:30:15.464234 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:15.464197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" event={"ID":"bfec6fe0-0408-4337-ae20-3424c0459db1","Type":"ContainerStarted","Data":"7d8c022bb26ce3ddf1206f6466712d7dd7ddc1ced975593b29f0b2d79c1a2b36"} Aug 01 00:30:15.464388 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:15.464253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" event={"ID":"bfec6fe0-0408-4337-ae20-3424c0459db1","Type":"ContainerStarted","Data":"4897e358394ff894d04460f653841084c1982b6bf9543a1220fcc80feb61bdc3"} Aug 01 00:30:15.567470 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:15.567433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:16.468856 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.468819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" event={"ID":"8a366acf-ae5b-4ed8-8fd6-5c7f892f6f79","Type":"ContainerStarted","Data":"82253507ed219983e8909585345295cba2b543904528e36b25fc8b607f150e76"} Aug 01 00:30:16.469327 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.468917 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:16.471110 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.471080 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" Aug 01 00:30:16.471419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.471386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" event={"ID":"bfec6fe0-0408-4337-ae20-3424c0459db1","Type":"ContainerStarted","Data":"cfe2bb57b7b946eeff0b7667b397211b50f015e08ca78b206db9a30910cd34ff"} Aug 01 00:30:16.485926 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.485876 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc45c5754-wdx4m" podStartSLOduration=1.255620538 podStartE2EDuration="8.485860435s" totalImagesPullingTime="7.230239897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:16.485037993 +0000 UTC m=+65.841829672" watchObservedRunningTime="2026-08-01 00:30:16.485860435 +0000 UTC m=+65.842652091" Aug 01 00:30:16.519699 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.519582 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dd74686-xqftr" podStartSLOduration=3.072865713 podStartE2EDuration="8.51956152s" totalImagesPullingTime="5.446695807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:16.518813145 +0000 UTC m=+65.875604804" watchObservedRunningTime="2026-08-01 00:30:16.51956152 +0000 UTC m=+65.876353176" Aug 01 00:30:16.579558 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.579517 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:16.855101 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.855008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:30:16.857585 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.857566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 01 00:30:16.867691 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.867665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e26feb7-e8a1-43f0-a20d-a18db3aa016b-original-pull-secret\") pod \"global-pull-secret-syncer-gkwsd\" (UID: \"6e26feb7-e8a1-43f0-a20d-a18db3aa016b\") " pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:30:16.879880 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.878898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkwsd" Aug 01 00:30:16.956425 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.956362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:30:16.958788 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.958766 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 01 00:30:16.969115 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:16.969085 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fd6a55-cd43-4cfd-8750-41c249783dd8-metrics-certs\") pod \"network-metrics-daemon-kkp8g\" (UID: \"84fd6a55-cd43-4cfd-8750-41c249783dd8\") " pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:30:17.022700 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.022666 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gkwsd"] Aug 01 00:30:17.026640 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:17.026608 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e26feb7_e8a1_43f0_a20d_a18db3aa016b.slice/crio-6ffd16950a537d7531e5c4ecf3cb4dca9fce639179b0b90fa6a2601fccd770cc WatchSource:0}: Error finding container 6ffd16950a537d7531e5c4ecf3cb4dca9fce639179b0b90fa6a2601fccd770cc: Status 404 returned error can't find the container with id 6ffd16950a537d7531e5c4ecf3cb4dca9fce639179b0b90fa6a2601fccd770cc Aug 01 00:30:17.057554 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.057520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:30:17.059911 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.059888 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 01 00:30:17.070310 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.070284 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 01 00:30:17.080658 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.080630 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmnwl\" (UniqueName: \"kubernetes.io/projected/234cbdba-5336-48c0-9814-fcc0cdad9272-kube-api-access-jmnwl\") pod \"network-check-target-p9lqk\" (UID: \"234cbdba-5336-48c0-9814-fcc0cdad9272\") " pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:30:17.173614 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.173581 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnzmh\"" Aug 01 00:30:17.181511 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.181488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:30:17.187123 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.187100 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c2x5h\"" Aug 01 00:30:17.195039 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.195015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkp8g" Aug 01 00:30:17.331822 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.331792 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p9lqk"] Aug 01 00:30:17.335718 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:17.335691 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234cbdba_5336_48c0_9814_fcc0cdad9272.slice/crio-159bb55cd55ad73d9212957c2015b92fa2745634ed55597c995bdb62c0778209 WatchSource:0}: Error finding container 159bb55cd55ad73d9212957c2015b92fa2745634ed55597c995bdb62c0778209: Status 404 returned error can't find the container with id 159bb55cd55ad73d9212957c2015b92fa2745634ed55597c995bdb62c0778209 Aug 01 00:30:17.348093 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.348065 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kkp8g"] Aug 01 00:30:17.351290 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:17.351265 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fd6a55_cd43_4cfd_8750_41c249783dd8.slice/crio-a6053f2c9f0816c51d03782c1fb5ec893b32e43837fb1f0a08c90354f84844ad WatchSource:0}: Error finding container a6053f2c9f0816c51d03782c1fb5ec893b32e43837fb1f0a08c90354f84844ad: Status 404 returned error can't find the container with id a6053f2c9f0816c51d03782c1fb5ec893b32e43837fb1f0a08c90354f84844ad Aug 01 00:30:17.476010 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.475912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gkwsd" event={"ID":"6e26feb7-e8a1-43f0-a20d-a18db3aa016b","Type":"ContainerStarted","Data":"6ffd16950a537d7531e5c4ecf3cb4dca9fce639179b0b90fa6a2601fccd770cc"} Aug 01 00:30:17.477171 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.477136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkp8g" event={"ID":"84fd6a55-cd43-4cfd-8750-41c249783dd8","Type":"ContainerStarted","Data":"a6053f2c9f0816c51d03782c1fb5ec893b32e43837fb1f0a08c90354f84844ad"} Aug 01 00:30:17.478201 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.478179 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p9lqk" event={"ID":"234cbdba-5336-48c0-9814-fcc0cdad9272","Type":"ContainerStarted","Data":"159bb55cd55ad73d9212957c2015b92fa2745634ed55597c995bdb62c0778209"} Aug 01 00:30:17.764019 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.763928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:30:17.766483 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:17.766457 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e119374b-fa39-4b71-8856-d50e943d4c4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v67t2\" (UID: \"e119374b-fa39-4b71-8856-d50e943d4c4c\") " pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:30:18.010726 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.010660 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9ldw2\"" Aug 01 00:30:18.018758 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.018648 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v67t2" Aug 01 00:30:18.049944 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.049895 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:18.148989 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.148924 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:18.185680 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.185484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v67t2"] Aug 01 00:30:18.200995 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.200907 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:18.391178 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:18.391075 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode119374b_fa39_4b71_8856_d50e943d4c4c.slice/crio-991e5fdb91ab77adaa6a90666589fe7fa1fccf4c8d250f383e85824e8d3932a2 WatchSource:0}: Error finding container 991e5fdb91ab77adaa6a90666589fe7fa1fccf4c8d250f383e85824e8d3932a2: Status 404 returned error can't find the container with id 991e5fdb91ab77adaa6a90666589fe7fa1fccf4c8d250f383e85824e8d3932a2 Aug 01 00:30:18.487552 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.487493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v67t2" event={"ID":"e119374b-fa39-4b71-8856-d50e943d4c4c","Type":"ContainerStarted","Data":"da57e3e0fcdcaed27d9f281b2db2e2cc16537f6bfa11206ff55251b0b0070517"} Aug 01 00:30:18.487934 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:18.487567 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v67t2" event={"ID":"e119374b-fa39-4b71-8856-d50e943d4c4c","Type":"ContainerStarted","Data":"991e5fdb91ab77adaa6a90666589fe7fa1fccf4c8d250f383e85824e8d3932a2"} Aug 01 00:30:19.494049 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:19.494003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkp8g" event={"ID":"84fd6a55-cd43-4cfd-8750-41c249783dd8","Type":"ContainerStarted","Data":"6c5451df35b982d5532616335ef52ff28f1963a1cf9408e5541f4a36a53ab6fa"} Aug 01 00:30:19.494568 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:19.494114 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkp8g" event={"ID":"84fd6a55-cd43-4cfd-8750-41c249783dd8","Type":"ContainerStarted","Data":"2331ff2aacd4aa71355fdd72e2d9b9cf586777e86be2aa617a689af13e5388e7"} Aug 01 00:30:19.509021 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:19.508712 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kkp8g" podStartSLOduration=67.437575578 podStartE2EDuration="1m8.508690111s" totalImagesPullingTime="1.071114533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:19.508462129 +0000 UTC m=+68.865253789" watchObservedRunningTime="2026-08-01 00:30:19.508690111 +0000 UTC m=+68.865481856" Aug 01 00:30:19.594001 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:19.593960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:19.649506 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:19.649465 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:20.961723 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:20.961681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:21.060423 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:21.060380 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:21.162537 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:21.162496 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:21.504619 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:21.504576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p9lqk" event={"ID":"234cbdba-5336-48c0-9814-fcc0cdad9272","Type":"ContainerStarted","Data":"1dff7a988d7e6173ed007b9ba2c67526237b980f08e025ae1150abfe7923c6cb"} Aug 01 00:30:21.504898 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:21.504675 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:30:21.506844 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:21.506814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v67t2" event={"ID":"e119374b-fa39-4b71-8856-d50e943d4c4c","Type":"ContainerStarted","Data":"16b98bb5b73eff887dce047388ce791b97a2d42695133227df1e7f52cabb3efe"} Aug 01 00:30:21.522265 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:21.522207 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p9lqk" podStartSLOduration=67.306469323 podStartE2EDuration="1m10.522189898s" totalImagesPullingTime="3.215720575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:21.521463864 +0000 UTC m=+70.878255532" watchObservedRunningTime="2026-08-01 00:30:21.522189898 +0000 UTC m=+70.878981553" Aug 01 00:30:22.397495 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:22.397451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:22.498616 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:22.498580 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:22.512605 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:22.512569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gkwsd" event={"ID":"6e26feb7-e8a1-43f0-a20d-a18db3aa016b","Type":"ContainerStarted","Data":"0cb273fe246a0f48ab3be8f1366dbb8ce8ca415d50b7bd30347d707e142377e0"} Aug 01 00:30:22.529809 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:22.529756 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gkwsd" podStartSLOduration=66.228594746 podStartE2EDuration="1m11.529737328s" totalImagesPullingTime="5.301142582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:22.528334987 +0000 UTC m=+71.885126643" watchObservedRunningTime="2026-08-01 00:30:22.529737328 +0000 UTC m=+71.886528985" Aug 01 00:30:22.987204 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:22.987136 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:23.213215 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.213170 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:23.277845 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.277762 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lsv64"] Aug 01 00:30:23.290258 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.290229 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.292938 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.292894 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 01 00:30:23.293097 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.292999 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 01 00:30:23.293097 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.293019 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x7kb2\"" Aug 01 00:30:23.293870 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.293851 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 01 00:30:23.294123 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.294097 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 01 00:30:23.294123 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.294116 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 01 00:30:23.294305 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.294287 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 01 00:30:23.296367 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.296342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:23.363073 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.363034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:23.414129 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414093 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l49q\" (UniqueName: \"kubernetes.io/projected/9932561e-11db-4dc2-856f-35bf72d17c5f-kube-api-access-7l49q\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414129 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414132 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-wtmp\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-textfile\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9932561e-11db-4dc2-856f-35bf72d17c5f-metrics-client-ca\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-root\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-sys\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.414676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.414377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-tls\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.470657 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.470616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:23.514887 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.514852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-textfile\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515070 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.514898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515070 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.514930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9932561e-11db-4dc2-856f-35bf72d17c5f-metrics-client-ca\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515070 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.514969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-root\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515070 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-sys\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515070 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-tls\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l49q\" (UniqueName: \"kubernetes.io/projected/9932561e-11db-4dc2-856f-35bf72d17c5f-kube-api-access-7l49q\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-root\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-sys\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-wtmp\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-textfile\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515447 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-wtmp\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.515796 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515741 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.516002 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.515978 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9932561e-11db-4dc2-856f-35bf72d17c5f-metrics-client-ca\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.517487 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.517458 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-tls\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.517581 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.517498 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9932561e-11db-4dc2-856f-35bf72d17c5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.523431 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.523412 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l49q\" (UniqueName: \"kubernetes.io/projected/9932561e-11db-4dc2-856f-35bf72d17c5f-kube-api-access-7l49q\") pod \"node-exporter-lsv64\" (UID: \"9932561e-11db-4dc2-856f-35bf72d17c5f\") " pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.575258 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.575226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:23.601236 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.601204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lsv64" Aug 01 00:30:23.634577 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:23.634548 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9932561e_11db_4dc2_856f_35bf72d17c5f.slice/crio-9297183fddfaa069091eee9b487d85c09c536c201dc28a5df280a3a5fca986bd WatchSource:0}: Error finding container 9297183fddfaa069091eee9b487d85c09c536c201dc28a5df280a3a5fca986bd: Status 404 returned error can't find the container with id 9297183fddfaa069091eee9b487d85c09c536c201dc28a5df280a3a5fca986bd Aug 01 00:30:23.663728 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.663675 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:23.850516 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:23.850414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:24.217721 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:24.217678 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:24.448596 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:24.448558 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:24.526732 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:24.526644 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v67t2" event={"ID":"e119374b-fa39-4b71-8856-d50e943d4c4c","Type":"ContainerStarted","Data":"b6cb2e8d6e07f1d1401eea8f24fbefabc402ed94b07978431178af90c0240c31"} Aug 01 00:30:24.527992 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:24.527957 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lsv64" event={"ID":"9932561e-11db-4dc2-856f-35bf72d17c5f","Type":"ContainerStarted","Data":"9297183fddfaa069091eee9b487d85c09c536c201dc28a5df280a3a5fca986bd"} Aug 01 00:30:24.545011 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:24.544955 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v67t2" podStartSLOduration=34.519451942 podStartE2EDuration="39.544933685s" totalImagesPullingTime="5.025481743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:29:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:24.54446976 +0000 UTC m=+73.901261417" watchObservedRunningTime="2026-08-01 00:30:24.544933685 +0000 UTC m=+73.901725343" Aug 01 00:30:24.548063 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:24.548015 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:24.828252 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:24.828145 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:25.532946 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:25.532914 2575 generic.go:356] "Generic (PLEG): container finished" podID="9932561e-11db-4dc2-856f-35bf72d17c5f" containerID="a2c7ab89552d1a30e4f55daa8a50ce76c2ec7706ddb92a74adc4948f8957c627" exitCode=0 Aug 01 00:30:25.533395 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:25.533006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lsv64" event={"ID":"9932561e-11db-4dc2-856f-35bf72d17c5f","Type":"ContainerDied","Data":"a2c7ab89552d1a30e4f55daa8a50ce76c2ec7706ddb92a74adc4948f8957c627"} Aug 01 00:30:25.564888 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:25.564853 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:25.620443 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:25.620408 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:26.147387 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.147350 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:26.208601 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.208555 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:26.267421 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.267386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:26.538900 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.538866 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lsv64" event={"ID":"9932561e-11db-4dc2-856f-35bf72d17c5f","Type":"ContainerStarted","Data":"ddb5c0dea1a2c87614cdf5d964cb77a8058c243104d779e22130d54e7f31750d"} Aug 01 00:30:26.539284 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.538952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lsv64" event={"ID":"9932561e-11db-4dc2-856f-35bf72d17c5f","Type":"ContainerStarted","Data":"6d104cb8bf05e654a5af2e3bdf372cf5b0433a22d2079d4773e2374d183aefaf"} Aug 01 00:30:26.558413 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.558368 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lsv64" podStartSLOduration=2.291449283 podStartE2EDuration="3.558351899s" totalImagesPullingTime="1.201643311s" totalInitContainerRuntime="65.259305ms" isStatefulPod=true podCreationTimestamp="2026-08-01 00:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:26.55801171 +0000 UTC m=+75.914803365" watchObservedRunningTime="2026-08-01 00:30:26.558351899 +0000 UTC m=+75.915143556" Aug 01 00:30:26.737726 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.737681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:26.837433 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.837362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:26.937600 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:26.937561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:28.008330 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.008296 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:28.102295 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.102234 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:28.205749 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.205711 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:28.495443 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.495414 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-769449954f-jdzdt"] Aug 01 00:30:28.498731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.498713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.501316 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.501288 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 01 00:30:28.501448 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.501294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 01 00:30:28.501448 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.501438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6wdm7\"" Aug 01 00:30:28.501562 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.501302 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 01 00:30:28.501562 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.501300 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 01 00:30:28.501562 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.501302 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 01 00:30:28.506753 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.506721 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 01 00:30:28.508800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.508779 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-769449954f-jdzdt"] Aug 01 00:30:28.549067 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-telemeter-client-tls\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.549271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-metrics-client-ca\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.549271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-federate-client-tls\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.549271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-serving-certs-ca-bundle\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.549271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.549271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9qj\" (UniqueName: \"kubernetes.io/projected/92ca8f50-32cc-4843-af2d-e6dae49639f3-kube-api-access-7l9qj\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.549436 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-secret-telemeter-client\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.549436 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.549314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650056 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-serving-certs-ca-bundle\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9qj\" (UniqueName: \"kubernetes.io/projected/92ca8f50-32cc-4843-af2d-e6dae49639f3-kube-api-access-7l9qj\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-secret-telemeter-client\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-telemeter-client-tls\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650474 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650211 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-metrics-client-ca\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650474 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-federate-client-tls\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.650852 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.650821 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-serving-certs-ca-bundle\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.651194 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.651149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-metrics-client-ca\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.651381 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.651356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ca8f50-32cc-4843-af2d-e6dae49639f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.652916 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.652885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.653007 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.652960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-secret-telemeter-client\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.653059 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.653038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-telemeter-client-tls\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.653095 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.653070 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92ca8f50-32cc-4843-af2d-e6dae49639f3-federate-client-tls\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.658374 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.658346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9qj\" (UniqueName: \"kubernetes.io/projected/92ca8f50-32cc-4843-af2d-e6dae49639f3-kube-api-access-7l9qj\") pod \"telemeter-client-769449954f-jdzdt\" (UID: \"92ca8f50-32cc-4843-af2d-e6dae49639f3\") " pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.808685 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.808584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" Aug 01 00:30:28.945904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:28.945861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-769449954f-jdzdt"] Aug 01 00:30:28.949530 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:30:28.949505 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ca8f50_32cc_4843_af2d_e6dae49639f3.slice/crio-a3550ceb10df11f0a6f603d821ebe2d05ebf17a80cd07ff2dc6b055c985479ad WatchSource:0}: Error finding container a3550ceb10df11f0a6f603d821ebe2d05ebf17a80cd07ff2dc6b055c985479ad: Status 404 returned error can't find the container with id a3550ceb10df11f0a6f603d821ebe2d05ebf17a80cd07ff2dc6b055c985479ad Aug 01 00:30:29.357521 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:29.357488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:29.449897 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:29.449824 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:29.518449 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:29.518414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:29.549661 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:29.549626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" event={"ID":"92ca8f50-32cc-4843-af2d-e6dae49639f3","Type":"ContainerStarted","Data":"a3550ceb10df11f0a6f603d821ebe2d05ebf17a80cd07ff2dc6b055c985479ad"} Aug 01 00:30:30.440050 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:30.440009 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:30.542291 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:30.542256 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:31.412357 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:31.412319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:31.480762 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:31.480705 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:31.558528 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:31.558489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" event={"ID":"92ca8f50-32cc-4843-af2d-e6dae49639f3","Type":"ContainerStarted","Data":"936fa7d274feca7c445d3ab32b6f5aaf9b7582ffce5fd85cd91e240d4916a82f"} Aug 01 00:30:32.170376 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:32.170337 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:32.564223 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:32.564187 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" event={"ID":"92ca8f50-32cc-4843-af2d-e6dae49639f3","Type":"ContainerStarted","Data":"4489f907e7e1795c3a76f627a856f0bbea2063c0c1b7fbae21600148715dd04d"} Aug 01 00:30:32.564600 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:32.564250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" event={"ID":"92ca8f50-32cc-4843-af2d-e6dae49639f3","Type":"ContainerStarted","Data":"1df92c4a27c16bb36dbba33c294eb1c44f7a050b7d6b14afe0e132a4cfee75df"} Aug 01 00:30:32.584830 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:32.584777 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-769449954f-jdzdt" podStartSLOduration=1.190475668 podStartE2EDuration="4.58475937s" totalImagesPullingTime="3.394283702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:32.584332042 +0000 UTC m=+81.941123700" watchObservedRunningTime="2026-08-01 00:30:32.58475937 +0000 UTC m=+81.941551026" Aug 01 00:30:33.181664 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:33.181624 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:40.304244 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:40.304208 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:40.404227 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:40.404186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:40.466369 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:40.466334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:42.105009 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:42.104966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:42.159924 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:42.159893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:43.384510 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:43.384467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:43.466693 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:43.466654 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:43.710619 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:43.710584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:43.768045 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:43.768010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:43.819887 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:43.819858 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:52.515716 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:52.515686 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p9lqk" Aug 01 00:30:55.319778 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:55.319739 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:30:55.370952 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:30:55.370908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:01.805298 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:01.805254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:01.874967 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:01.874927 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:01.938036 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:01.937995 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:04.750120 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:04.750082 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:04.846926 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:04.846891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:39.297442 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:39.297402 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:39.349828 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:39.349789 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:51.295287 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:51.295248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:51.344096 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:51.344064 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:55.307140 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:55.307109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:31:55.408523 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:31:55.408486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:01.110152 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:01.110117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:01.171558 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:01.171518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:12.071246 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:12.071205 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:12.151694 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:12.151653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:33.715096 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:33.715063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:33.811769 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:33.811734 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:34.286587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.286557 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk"] Aug 01 00:32:34.289590 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.289573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.292249 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.292228 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:32:34.292979 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.292966 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-844wz\"" Aug 01 00:32:34.293075 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.292970 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:32:34.299808 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.299777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk"] Aug 01 00:32:34.443753 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.443715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.443753 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.443757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.443981 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.443783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dmc\" (UniqueName: \"kubernetes.io/projected/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-kube-api-access-q4dmc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.544910 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.544810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.544910 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.544853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.544910 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.544877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dmc\" (UniqueName: \"kubernetes.io/projected/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-kube-api-access-q4dmc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.545317 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.545293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.545376 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.545315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.552933 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.552905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dmc\" (UniqueName: \"kubernetes.io/projected/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-kube-api-access-q4dmc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.597899 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.597862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:34.727633 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.727598 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk"] Aug 01 00:32:34.730599 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:32:34.730571 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc15a9d_cfc1_443f_8901_b6bb7d7a493d.slice/crio-df9ecfe7bddacbcbcb5bf0e00c6b5f745d287f858d14b89b3bf67eadaea277e4 WatchSource:0}: Error finding container df9ecfe7bddacbcbcb5bf0e00c6b5f745d287f858d14b89b3bf67eadaea277e4: Status 404 returned error can't find the container with id df9ecfe7bddacbcbcb5bf0e00c6b5f745d287f858d14b89b3bf67eadaea277e4 Aug 01 00:32:34.977578 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:34.977545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" event={"ID":"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d","Type":"ContainerStarted","Data":"df9ecfe7bddacbcbcb5bf0e00c6b5f745d287f858d14b89b3bf67eadaea277e4"} Aug 01 00:32:36.292063 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:36.292027 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:36.343027 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:36.342988 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:40.336437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:40.336389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:40.384640 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:40.384541 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:42.007972 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:42.007932 2575 generic.go:356] "Generic (PLEG): container finished" podID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerID="b443d88a24f6b8b6174cd904d7d3f2469e98d126d300e2b5e3496e3ad092a8fc" exitCode=0 Aug 01 00:32:42.008383 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:42.008018 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" event={"ID":"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d","Type":"ContainerDied","Data":"b443d88a24f6b8b6174cd904d7d3f2469e98d126d300e2b5e3496e3ad092a8fc"} Aug 01 00:32:44.017096 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:44.017062 2575 generic.go:356] "Generic (PLEG): container finished" podID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerID="b7c695ede6f2c7ab41df79dc68ce93770ad13d5caeda364ea11fd32972c003c1" exitCode=0 Aug 01 00:32:44.017526 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:44.017109 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" event={"ID":"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d","Type":"ContainerDied","Data":"b7c695ede6f2c7ab41df79dc68ce93770ad13d5caeda364ea11fd32972c003c1"} Aug 01 00:32:52.052174 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:52.052122 2575 generic.go:356] "Generic (PLEG): container finished" podID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerID="d317c911e6ac4017c7e40d071703c7e119f4fb06b8aaecb03bed522971d1c40a" exitCode=0 Aug 01 00:32:52.052582 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:52.052198 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" event={"ID":"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d","Type":"ContainerDied","Data":"d317c911e6ac4017c7e40d071703c7e119f4fb06b8aaecb03bed522971d1c40a"} Aug 01 00:32:53.175930 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.175902 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:53.296900 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.296838 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-bundle\") pod \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " Aug 01 00:32:53.296900 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.296906 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4dmc\" (UniqueName: \"kubernetes.io/projected/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-kube-api-access-q4dmc\") pod \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " Aug 01 00:32:53.297129 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.296940 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-util\") pod \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\" (UID: \"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d\") " Aug 01 00:32:53.297601 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.297577 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-bundle" (OuterVolumeSpecName: "bundle") pod "3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" (UID: "3cc15a9d-cfc1-443f-8901-b6bb7d7a493d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:32:53.299265 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.299238 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-kube-api-access-q4dmc" (OuterVolumeSpecName: "kube-api-access-q4dmc") pod "3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" (UID: "3cc15a9d-cfc1-443f-8901-b6bb7d7a493d"). InnerVolumeSpecName "kube-api-access-q4dmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:32:53.301698 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.301667 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-util" (OuterVolumeSpecName: "util") pod "3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" (UID: "3cc15a9d-cfc1-443f-8901-b6bb7d7a493d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:32:53.397848 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.397750 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:32:53.397848 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.397810 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:32:53.397848 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:53.397821 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4dmc\" (UniqueName: \"kubernetes.io/projected/3cc15a9d-cfc1-443f-8901-b6bb7d7a493d-kube-api-access-q4dmc\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:32:54.060739 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:54.060705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" event={"ID":"3cc15a9d-cfc1-443f-8901-b6bb7d7a493d","Type":"ContainerDied","Data":"df9ecfe7bddacbcbcb5bf0e00c6b5f745d287f858d14b89b3bf67eadaea277e4"} Aug 01 00:32:54.060890 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:54.060757 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9ecfe7bddacbcbcb5bf0e00c6b5f745d287f858d14b89b3bf67eadaea277e4" Aug 01 00:32:54.060890 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:54.060714 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knnpk" Aug 01 00:32:55.253375 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:55.253340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:56.033539 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:56.033506 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:58.668140 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:58.668102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:32:58.769789 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:32:58.769750 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:01.082585 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.082542 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:01.154764 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.154729 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:01.519754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.519718 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:01.626327 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.626297 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:01.836463 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836381 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9"] Aug 01 00:33:01.836678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836626 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerName="util" Aug 01 00:33:01.836678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836637 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerName="util" Aug 01 00:33:01.836678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836644 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerName="pull" Aug 01 00:33:01.836678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836649 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerName="pull" Aug 01 00:33:01.836678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836661 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerName="extract" Aug 01 00:33:01.836678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836667 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerName="extract" Aug 01 00:33:01.836861 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.836714 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc15a9d-cfc1-443f-8901-b6bb7d7a493d" containerName="extract" Aug 01 00:33:01.839550 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.839530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.842118 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.842092 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-844wz\"" Aug 01 00:33:01.842291 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.842203 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:33:01.843021 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.843006 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:33:01.851452 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.851425 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9"] Aug 01 00:33:01.858321 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.858291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.858492 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.858350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhbgp\" (UniqueName: \"kubernetes.io/projected/751ef767-6428-4b83-97fc-e87e4a0ae9b2-kube-api-access-bhbgp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.858492 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.858450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.958797 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.958761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.958797 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.958798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.959061 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.958833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhbgp\" (UniqueName: \"kubernetes.io/projected/751ef767-6428-4b83-97fc-e87e4a0ae9b2-kube-api-access-bhbgp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.959327 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.959302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.959411 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.959357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:01.967644 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:01.967612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhbgp\" (UniqueName: \"kubernetes.io/projected/751ef767-6428-4b83-97fc-e87e4a0ae9b2-kube-api-access-bhbgp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:02.148896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:02.148806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:02.279995 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:02.279874 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9"] Aug 01 00:33:02.282530 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:33:02.282507 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751ef767_6428_4b83_97fc_e87e4a0ae9b2.slice/crio-386120462d5325805e27187d05b8292eb09cd07476a483d87445848fb56deec7 WatchSource:0}: Error finding container 386120462d5325805e27187d05b8292eb09cd07476a483d87445848fb56deec7: Status 404 returned error can't find the container with id 386120462d5325805e27187d05b8292eb09cd07476a483d87445848fb56deec7 Aug 01 00:33:02.584912 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:02.584854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:02.685020 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:02.684990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:03.087747 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.087717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:03.091575 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.091543 2575 generic.go:356] "Generic (PLEG): container finished" podID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerID="dc3c39e613fef6d2ce8ea971e8c85b543a06c3f714f22febd5ce945e7b45d4b3" exitCode=0 Aug 01 00:33:03.091747 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.091625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" event={"ID":"751ef767-6428-4b83-97fc-e87e4a0ae9b2","Type":"ContainerDied","Data":"dc3c39e613fef6d2ce8ea971e8c85b543a06c3f714f22febd5ce945e7b45d4b3"} Aug 01 00:33:03.091747 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.091675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" event={"ID":"751ef767-6428-4b83-97fc-e87e4a0ae9b2","Type":"ContainerStarted","Data":"386120462d5325805e27187d05b8292eb09cd07476a483d87445848fb56deec7"} Aug 01 00:33:03.465714 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.465677 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:03.867793 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.867718 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tgh59"] Aug 01 00:33:03.870794 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.870776 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:03.873541 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.873518 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sc4ld\"" Aug 01 00:33:03.873643 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.873550 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 01 00:33:03.874374 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.874358 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 01 00:33:03.880498 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.880476 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tgh59"] Aug 01 00:33:03.974577 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.974539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blsbt\" (UniqueName: \"kubernetes.io/projected/afaa4164-6e3a-44da-a988-6ee783aa5114-kube-api-access-blsbt\") pod \"cert-manager-cainjector-559cdd8c87-tgh59\" (UID: \"afaa4164-6e3a-44da-a988-6ee783aa5114\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:03.974764 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:03.974597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afaa4164-6e3a-44da-a988-6ee783aa5114-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tgh59\" (UID: \"afaa4164-6e3a-44da-a988-6ee783aa5114\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:04.075107 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:04.075064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blsbt\" (UniqueName: \"kubernetes.io/projected/afaa4164-6e3a-44da-a988-6ee783aa5114-kube-api-access-blsbt\") pod \"cert-manager-cainjector-559cdd8c87-tgh59\" (UID: \"afaa4164-6e3a-44da-a988-6ee783aa5114\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:04.075275 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:04.075122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afaa4164-6e3a-44da-a988-6ee783aa5114-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tgh59\" (UID: \"afaa4164-6e3a-44da-a988-6ee783aa5114\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:04.083388 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:04.083355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afaa4164-6e3a-44da-a988-6ee783aa5114-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tgh59\" (UID: \"afaa4164-6e3a-44da-a988-6ee783aa5114\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:04.083516 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:04.083496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blsbt\" (UniqueName: \"kubernetes.io/projected/afaa4164-6e3a-44da-a988-6ee783aa5114-kube-api-access-blsbt\") pod \"cert-manager-cainjector-559cdd8c87-tgh59\" (UID: \"afaa4164-6e3a-44da-a988-6ee783aa5114\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:04.179403 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:04.179370 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" Aug 01 00:33:04.330322 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:04.330286 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tgh59"] Aug 01 00:33:04.333407 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:33:04.333381 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafaa4164_6e3a_44da_a988_6ee783aa5114.slice/crio-57dc69da3b90144b5fb43f25f503c571d4e98ee22e64d1acba802f58a01b443c WatchSource:0}: Error finding container 57dc69da3b90144b5fb43f25f503c571d4e98ee22e64d1acba802f58a01b443c: Status 404 returned error can't find the container with id 57dc69da3b90144b5fb43f25f503c571d4e98ee22e64d1acba802f58a01b443c Aug 01 00:33:05.100237 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:05.100196 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" event={"ID":"afaa4164-6e3a-44da-a988-6ee783aa5114","Type":"ContainerStarted","Data":"57dc69da3b90144b5fb43f25f503c571d4e98ee22e64d1acba802f58a01b443c"} Aug 01 00:33:05.459789 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:05.459757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:05.561564 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:05.561525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:06.105127 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:06.105088 2575 generic.go:356] "Generic (PLEG): container finished" podID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerID="093d3e68622440c5b81293391c29899b7fa3d16fc985493dc3117c6358d21759" exitCode=0 Aug 01 00:33:06.105515 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:06.105152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" event={"ID":"751ef767-6428-4b83-97fc-e87e4a0ae9b2","Type":"ContainerDied","Data":"093d3e68622440c5b81293391c29899b7fa3d16fc985493dc3117c6358d21759"} Aug 01 00:33:07.110460 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:07.110425 2575 generic.go:356] "Generic (PLEG): container finished" podID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerID="54d48647102f55fff0b80843aeeb5654fae151e310095419cb15b273e5f697b2" exitCode=0 Aug 01 00:33:07.110866 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:07.110488 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" event={"ID":"751ef767-6428-4b83-97fc-e87e4a0ae9b2","Type":"ContainerDied","Data":"54d48647102f55fff0b80843aeeb5654fae151e310095419cb15b273e5f697b2"} Aug 01 00:33:08.074090 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.074059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:08.239088 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.239061 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:08.306551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.306511 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhbgp\" (UniqueName: \"kubernetes.io/projected/751ef767-6428-4b83-97fc-e87e4a0ae9b2-kube-api-access-bhbgp\") pod \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " Aug 01 00:33:08.306551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.306562 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-util\") pod \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " Aug 01 00:33:08.306806 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.306594 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-bundle\") pod \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\" (UID: \"751ef767-6428-4b83-97fc-e87e4a0ae9b2\") " Aug 01 00:33:08.307006 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.306979 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-bundle" (OuterVolumeSpecName: "bundle") pod "751ef767-6428-4b83-97fc-e87e4a0ae9b2" (UID: "751ef767-6428-4b83-97fc-e87e4a0ae9b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:33:08.308651 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.308626 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751ef767-6428-4b83-97fc-e87e4a0ae9b2-kube-api-access-bhbgp" (OuterVolumeSpecName: "kube-api-access-bhbgp") pod "751ef767-6428-4b83-97fc-e87e4a0ae9b2" (UID: "751ef767-6428-4b83-97fc-e87e4a0ae9b2"). InnerVolumeSpecName "kube-api-access-bhbgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:33:08.311545 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.311517 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-util" (OuterVolumeSpecName: "util") pod "751ef767-6428-4b83-97fc-e87e4a0ae9b2" (UID: "751ef767-6428-4b83-97fc-e87e4a0ae9b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:33:08.407790 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.407761 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:33:08.407790 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.407787 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/751ef767-6428-4b83-97fc-e87e4a0ae9b2-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:33:08.407985 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:08.407796 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhbgp\" (UniqueName: \"kubernetes.io/projected/751ef767-6428-4b83-97fc-e87e4a0ae9b2-kube-api-access-bhbgp\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:33:09.118980 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:09.118945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" event={"ID":"751ef767-6428-4b83-97fc-e87e4a0ae9b2","Type":"ContainerDied","Data":"386120462d5325805e27187d05b8292eb09cd07476a483d87445848fb56deec7"} Aug 01 00:33:09.118980 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:09.118968 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2hrj9" Aug 01 00:33:09.119212 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:09.119043 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="386120462d5325805e27187d05b8292eb09cd07476a483d87445848fb56deec7" Aug 01 00:33:11.518822 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:11.518778 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:11.568354 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:11.568301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:12.277977 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:12.277944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:15.082231 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:15.082195 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:15.142896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:15.142853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" event={"ID":"afaa4164-6e3a-44da-a988-6ee783aa5114","Type":"ContainerStarted","Data":"0976643c40f196f38e143f57b0bead44ff2b0a4d5e94423240490be6a1ddbe4c"} Aug 01 00:33:15.159002 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:15.158944 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tgh59" podStartSLOduration=1.865350281 podStartE2EDuration="12.158927701s" totalImagesPullingTime="10.29357742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:15.158089748 +0000 UTC m=+244.514881405" watchObservedRunningTime="2026-08-01 00:33:15.158927701 +0000 UTC m=+244.515719357" Aug 01 00:33:15.162826 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:15.162796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:15.904245 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:15.904210 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:15.957938 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:15.957904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:20.876947 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:20.876910 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:20.980291 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:20.980258 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:22.544554 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.544516 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh"] Aug 01 00:33:22.545024 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.544862 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerName="pull" Aug 01 00:33:22.545024 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.544879 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerName="pull" Aug 01 00:33:22.545024 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.544894 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerName="util" Aug 01 00:33:22.545024 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.544902 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerName="util" Aug 01 00:33:22.545024 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.544911 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerName="extract" Aug 01 00:33:22.545024 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.544920 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerName="extract" Aug 01 00:33:22.545024 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.545015 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="751ef767-6428-4b83-97fc-e87e4a0ae9b2" containerName="extract" Aug 01 00:33:22.551854 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.551818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.554353 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.554331 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:33:22.554707 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.554688 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:33:22.555400 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.555378 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-844wz\"" Aug 01 00:33:22.555585 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.555563 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh"] Aug 01 00:33:22.610907 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.610863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.611102 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.610914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.611102 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.610944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2m8p\" (UniqueName: \"kubernetes.io/projected/c6230427-b289-475b-9322-0da1a5245d28-kube-api-access-k2m8p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.688986 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.688945 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:22.712088 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.712057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.712206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.712101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.712206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.712129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2m8p\" (UniqueName: \"kubernetes.io/projected/c6230427-b289-475b-9322-0da1a5245d28-kube-api-access-k2m8p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.712505 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.712485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.712545 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.712497 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.719911 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.719872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2m8p\" (UniqueName: \"kubernetes.io/projected/c6230427-b289-475b-9322-0da1a5245d28-kube-api-access-k2m8p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.862728 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.862629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:22.995774 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:22.995741 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh"] Aug 01 00:33:22.999108 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:33:22.999079 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6230427_b289_475b_9322_0da1a5245d28.slice/crio-c2ab5cbbfc62e30005e84584e560811ab99ff1d4b15f29ac0f59b33163b4d260 WatchSource:0}: Error finding container c2ab5cbbfc62e30005e84584e560811ab99ff1d4b15f29ac0f59b33163b4d260: Status 404 returned error can't find the container with id c2ab5cbbfc62e30005e84584e560811ab99ff1d4b15f29ac0f59b33163b4d260 Aug 01 00:33:23.006812 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:23.006783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:23.173875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:23.173840 2575 generic.go:356] "Generic (PLEG): container finished" podID="c6230427-b289-475b-9322-0da1a5245d28" containerID="2b8048d35930584c0e37178d4e642eca5e87b33a0dd09c492b217a402f3bb941" exitCode=0 Aug 01 00:33:23.174043 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:23.173898 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" event={"ID":"c6230427-b289-475b-9322-0da1a5245d28","Type":"ContainerDied","Data":"2b8048d35930584c0e37178d4e642eca5e87b33a0dd09c492b217a402f3bb941"} Aug 01 00:33:23.174043 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:23.173934 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" event={"ID":"c6230427-b289-475b-9322-0da1a5245d28","Type":"ContainerStarted","Data":"c2ab5cbbfc62e30005e84584e560811ab99ff1d4b15f29ac0f59b33163b4d260"} Aug 01 00:33:24.178907 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:24.178872 2575 generic.go:356] "Generic (PLEG): container finished" podID="c6230427-b289-475b-9322-0da1a5245d28" containerID="cae87dd63cc8103284388a0562b0910c51a4230612a640f1df19f58712380789" exitCode=0 Aug 01 00:33:24.179307 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:24.178917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" event={"ID":"c6230427-b289-475b-9322-0da1a5245d28","Type":"ContainerDied","Data":"cae87dd63cc8103284388a0562b0910c51a4230612a640f1df19f58712380789"} Aug 01 00:33:25.184863 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:25.184829 2575 generic.go:356] "Generic (PLEG): container finished" podID="c6230427-b289-475b-9322-0da1a5245d28" containerID="538648847b7ba6b5a9445d4e5ce450d1cf869d4a4ebf0f78779b3b1466a6a711" exitCode=0 Aug 01 00:33:25.185335 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:25.184896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" event={"ID":"c6230427-b289-475b-9322-0da1a5245d28","Type":"ContainerDied","Data":"538648847b7ba6b5a9445d4e5ce450d1cf869d4a4ebf0f78779b3b1466a6a711"} Aug 01 00:33:26.324623 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.324589 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:26.439348 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.439254 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-bundle\") pod \"c6230427-b289-475b-9322-0da1a5245d28\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " Aug 01 00:33:26.439348 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.439295 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2m8p\" (UniqueName: \"kubernetes.io/projected/c6230427-b289-475b-9322-0da1a5245d28-kube-api-access-k2m8p\") pod \"c6230427-b289-475b-9322-0da1a5245d28\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " Aug 01 00:33:26.439348 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.439332 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-util\") pod \"c6230427-b289-475b-9322-0da1a5245d28\" (UID: \"c6230427-b289-475b-9322-0da1a5245d28\") " Aug 01 00:33:26.440226 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.440185 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-bundle" (OuterVolumeSpecName: "bundle") pod "c6230427-b289-475b-9322-0da1a5245d28" (UID: "c6230427-b289-475b-9322-0da1a5245d28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:33:26.441807 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.441772 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6230427-b289-475b-9322-0da1a5245d28-kube-api-access-k2m8p" (OuterVolumeSpecName: "kube-api-access-k2m8p") pod "c6230427-b289-475b-9322-0da1a5245d28" (UID: "c6230427-b289-475b-9322-0da1a5245d28"). InnerVolumeSpecName "kube-api-access-k2m8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:33:26.445596 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.445558 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-util" (OuterVolumeSpecName: "util") pod "c6230427-b289-475b-9322-0da1a5245d28" (UID: "c6230427-b289-475b-9322-0da1a5245d28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:33:26.539731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.539686 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:33:26.539731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.539718 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2m8p\" (UniqueName: \"kubernetes.io/projected/c6230427-b289-475b-9322-0da1a5245d28-kube-api-access-k2m8p\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:33:26.539731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:26.539728 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6230427-b289-475b-9322-0da1a5245d28-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:33:27.193938 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:27.193910 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" Aug 01 00:33:27.194124 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:27.193938 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjm8xh" event={"ID":"c6230427-b289-475b-9322-0da1a5245d28","Type":"ContainerDied","Data":"c2ab5cbbfc62e30005e84584e560811ab99ff1d4b15f29ac0f59b33163b4d260"} Aug 01 00:33:27.194124 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:27.193982 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ab5cbbfc62e30005e84584e560811ab99ff1d4b15f29ac0f59b33163b4d260" Aug 01 00:33:28.644688 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:28.644646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:34.893392 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:34.893288 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:34.994941 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:34.994904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:35.495123 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.495087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:35.726408 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.726372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:35.836783 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.836691 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:35.866274 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866244 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd"] Aug 01 00:33:35.866498 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866486 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6230427-b289-475b-9322-0da1a5245d28" containerName="pull" Aug 01 00:33:35.866555 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866499 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6230427-b289-475b-9322-0da1a5245d28" containerName="pull" Aug 01 00:33:35.866555 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866510 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6230427-b289-475b-9322-0da1a5245d28" containerName="extract" Aug 01 00:33:35.866555 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866516 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6230427-b289-475b-9322-0da1a5245d28" containerName="extract" Aug 01 00:33:35.866555 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866546 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6230427-b289-475b-9322-0da1a5245d28" containerName="util" Aug 01 00:33:35.866555 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866552 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6230427-b289-475b-9322-0da1a5245d28" containerName="util" Aug 01 00:33:35.866713 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.866593 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6230427-b289-475b-9322-0da1a5245d28" containerName="extract" Aug 01 00:33:35.868572 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.868552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:35.871852 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.871829 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 01 00:33:35.872086 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.872069 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 01 00:33:35.872470 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.872444 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 01 00:33:35.872587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.872484 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 01 00:33:35.872587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.872511 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8f8\"" Aug 01 00:33:35.884327 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:35.884296 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd"] Aug 01 00:33:36.005399 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.005352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/976f6905-bc5f-4738-8453-ff6bb26b2c32-manifests\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.005399 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.005402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/976f6905-bc5f-4738-8453-ff6bb26b2c32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.005834 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.005426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7h5b\" (UniqueName: \"kubernetes.io/projected/976f6905-bc5f-4738-8453-ff6bb26b2c32-kube-api-access-w7h5b\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.005834 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.005473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/976f6905-bc5f-4738-8453-ff6bb26b2c32-tmp\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.005834 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.005592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/976f6905-bc5f-4738-8453-ff6bb26b2c32-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.015437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.015410 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:36.106656 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.106558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/976f6905-bc5f-4738-8453-ff6bb26b2c32-manifests\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.106656 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.106604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/976f6905-bc5f-4738-8453-ff6bb26b2c32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.106656 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.106627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7h5b\" (UniqueName: \"kubernetes.io/projected/976f6905-bc5f-4738-8453-ff6bb26b2c32-kube-api-access-w7h5b\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.106656 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.106653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/976f6905-bc5f-4738-8453-ff6bb26b2c32-tmp\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.106995 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.106699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/976f6905-bc5f-4738-8453-ff6bb26b2c32-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.107060 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.107036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/976f6905-bc5f-4738-8453-ff6bb26b2c32-manifests\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.109047 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.109023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/976f6905-bc5f-4738-8453-ff6bb26b2c32-tmp\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.109243 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.109222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/976f6905-bc5f-4738-8453-ff6bb26b2c32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.109401 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.109380 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/976f6905-bc5f-4738-8453-ff6bb26b2c32-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.110472 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.110448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:36.114676 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.114657 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7h5b\" (UniqueName: \"kubernetes.io/projected/976f6905-bc5f-4738-8453-ff6bb26b2c32-kube-api-access-w7h5b\") pod \"opendatahub-operator-controller-manager-6c56689d88-mtpkd\" (UID: \"976f6905-bc5f-4738-8453-ff6bb26b2c32\") " pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.178549 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.178512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:36.210446 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.210412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:36.313572 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.313439 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:36.313703 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.313677 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd"] Aug 01 00:33:36.317640 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:33:36.317608 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976f6905_bc5f_4738_8453_ff6bb26b2c32.slice/crio-eb29a4aaff11389c90ac67808a69f19ff002a1af4188a2307543d3d7cdca05ee WatchSource:0}: Error finding container eb29a4aaff11389c90ac67808a69f19ff002a1af4188a2307543d3d7cdca05ee: Status 404 returned error can't find the container with id eb29a4aaff11389c90ac67808a69f19ff002a1af4188a2307543d3d7cdca05ee Aug 01 00:33:36.410625 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:36.410574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:37.231964 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:37.231925 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" event={"ID":"976f6905-bc5f-4738-8453-ff6bb26b2c32","Type":"ContainerStarted","Data":"eb29a4aaff11389c90ac67808a69f19ff002a1af4188a2307543d3d7cdca05ee"} Aug 01 00:33:40.245111 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:40.245074 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" event={"ID":"976f6905-bc5f-4738-8453-ff6bb26b2c32","Type":"ContainerStarted","Data":"ad97e7f93ed5f4a8d192f8d1159fd8b847da5443db0366812c3221fc150ea696"} Aug 01 00:33:40.928199 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:40.928142 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:41.028708 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:41.028666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:41.251102 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:41.251011 2575 generic.go:356] "Generic (PLEG): container finished" podID="976f6905-bc5f-4738-8453-ff6bb26b2c32" containerID="ad97e7f93ed5f4a8d192f8d1159fd8b847da5443db0366812c3221fc150ea696" exitCode=0 Aug 01 00:33:41.251102 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:41.251087 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" event={"ID":"976f6905-bc5f-4738-8453-ff6bb26b2c32","Type":"ContainerDied","Data":"ad97e7f93ed5f4a8d192f8d1159fd8b847da5443db0366812c3221fc150ea696"} Aug 01 00:33:42.257647 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:42.257614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" event={"ID":"976f6905-bc5f-4738-8453-ff6bb26b2c32","Type":"ContainerStarted","Data":"e6ce812dde94369a6457513df2804cdd6fc232f9d2d563d2899ef45ad61b5d44"} Aug 01 00:33:42.258064 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:42.257742 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:42.296401 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:42.296343 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" podStartSLOduration=2.864380133 podStartE2EDuration="7.296326387s" totalImagesPullingTime="3.371974119s" totalInitContainerRuntime="1.059972135s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:42.295564623 +0000 UTC m=+271.652356281" watchObservedRunningTime="2026-08-01 00:33:42.296326387 +0000 UTC m=+271.653118043" Aug 01 00:33:53.263818 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:53.263782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c56689d88-mtpkd" Aug 01 00:33:54.850243 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:54.850205 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:54.989851 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:54.989817 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:55.297711 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.297668 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:55.363531 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.363499 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99"] Aug 01 00:33:55.370965 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.370939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.373683 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.373651 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 01 00:33:55.373683 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.373663 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-29542\"" Aug 01 00:33:55.373905 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.373709 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 01 00:33:55.374942 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.374917 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 01 00:33:55.374942 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.374931 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 01 00:33:55.383972 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.383949 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99"] Aug 01 00:33:55.462683 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.462652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcmj\" (UniqueName: \"kubernetes.io/projected/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-kube-api-access-xzcmj\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.462683 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.462687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tmp\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.462918 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.462730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tls-certs\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.548378 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.548299 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m"] Aug 01 00:33:55.551388 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.551371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.554056 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.554033 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 01 00:33:55.554208 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.554080 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 01 00:33:55.554208 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.554059 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 01 00:33:55.554208 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.554103 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-t6hqt\"" Aug 01 00:33:55.554208 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.554046 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 01 00:33:55.554208 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.554079 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 01 00:33:55.563308 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.563281 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m"] Aug 01 00:33:55.563434 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.563391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcmj\" (UniqueName: \"kubernetes.io/projected/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-kube-api-access-xzcmj\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.563434 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.563415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tmp\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.563534 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.563446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tls-certs\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.563581 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:33:55.563533 2575 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Aug 01 00:33:55.563634 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:33:55.563600 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tls-certs podName:db9fd642-bcbd-487e-95ef-5e3aba0cf39b nodeName:}" failed. No retries permitted until 2026-08-01 00:33:56.063580021 +0000 UTC m=+285.420371660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tls-certs") pod "kube-auth-proxy-6db59d85d8-rtr99" (UID: "db9fd642-bcbd-487e-95ef-5e3aba0cf39b") : secret "kube-auth-proxy-tls" not found Aug 01 00:33:55.565698 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.565672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tmp\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.583051 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.583015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcmj\" (UniqueName: \"kubernetes.io/projected/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-kube-api-access-xzcmj\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:55.653269 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.653230 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:55.664149 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.664117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c0a414bd-65cf-48c7-992f-a0a21e6620aa-manager-config\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.664307 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.664179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjwg\" (UniqueName: \"kubernetes.io/projected/c0a414bd-65cf-48c7-992f-a0a21e6620aa-kube-api-access-dcjwg\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.664307 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.664219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0a414bd-65cf-48c7-992f-a0a21e6620aa-metrics-cert\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.664307 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.664279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0a414bd-65cf-48c7-992f-a0a21e6620aa-cert\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.765096 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.765061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c0a414bd-65cf-48c7-992f-a0a21e6620aa-manager-config\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.765096 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.765103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjwg\" (UniqueName: \"kubernetes.io/projected/c0a414bd-65cf-48c7-992f-a0a21e6620aa-kube-api-access-dcjwg\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.765391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.765130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0a414bd-65cf-48c7-992f-a0a21e6620aa-metrics-cert\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.765391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.765235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0a414bd-65cf-48c7-992f-a0a21e6620aa-cert\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.765845 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.765818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c0a414bd-65cf-48c7-992f-a0a21e6620aa-manager-config\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.767731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.767707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0a414bd-65cf-48c7-992f-a0a21e6620aa-metrics-cert\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.767830 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.767747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0a414bd-65cf-48c7-992f-a0a21e6620aa-cert\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.777365 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.777335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjwg\" (UniqueName: \"kubernetes.io/projected/c0a414bd-65cf-48c7-992f-a0a21e6620aa-kube-api-access-dcjwg\") pod \"lws-controller-manager-5dd57cb99d-9g67m\" (UID: \"c0a414bd-65cf-48c7-992f-a0a21e6620aa\") " pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.860484 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.860379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:55.862182 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.862140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:55.962361 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.962329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:55.996077 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:55.995996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m"] Aug 01 00:33:55.998342 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:33:55.998308 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a414bd_65cf_48c7_992f_a0a21e6620aa.slice/crio-8a6e45b35a1b3eed2f4c5b187d8920f4d841267b15d22e696e25ab649b5e9bd4 WatchSource:0}: Error finding container 8a6e45b35a1b3eed2f4c5b187d8920f4d841267b15d22e696e25ab649b5e9bd4: Status 404 returned error can't find the container with id 8a6e45b35a1b3eed2f4c5b187d8920f4d841267b15d22e696e25ab649b5e9bd4 Aug 01 00:33:56.067519 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.067487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tls-certs\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:56.069918 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.069887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db9fd642-bcbd-487e-95ef-5e3aba0cf39b-tls-certs\") pod \"kube-auth-proxy-6db59d85d8-rtr99\" (UID: \"db9fd642-bcbd-487e-95ef-5e3aba0cf39b\") " pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:56.121472 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.121373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:56.280866 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.280832 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" Aug 01 00:33:56.316653 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.316622 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" event={"ID":"c0a414bd-65cf-48c7-992f-a0a21e6620aa","Type":"ContainerStarted","Data":"8a6e45b35a1b3eed2f4c5b187d8920f4d841267b15d22e696e25ab649b5e9bd4"} Aug 01 00:33:56.387557 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.387464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:56.425824 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.425798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99"] Aug 01 00:33:56.427640 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:33:56.427614 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9fd642_bcbd_487e_95ef_5e3aba0cf39b.slice/crio-4a550c4fc889bd635f430eef4e136962d4990341a6e99991b09d55547529fbed WatchSource:0}: Error finding container 4a550c4fc889bd635f430eef4e136962d4990341a6e99991b09d55547529fbed: Status 404 returned error can't find the container with id 4a550c4fc889bd635f430eef4e136962d4990341a6e99991b09d55547529fbed Aug 01 00:33:56.621043 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.621007 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:56.717386 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:56.717349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:57.326204 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:57.326168 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" event={"ID":"db9fd642-bcbd-487e-95ef-5e3aba0cf39b","Type":"ContainerStarted","Data":"4a550c4fc889bd635f430eef4e136962d4990341a6e99991b09d55547529fbed"} Aug 01 00:33:57.663795 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:57.663756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:33:58.334018 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:58.333974 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" event={"ID":"c0a414bd-65cf-48c7-992f-a0a21e6620aa","Type":"ContainerStarted","Data":"f3f5c503959195ab4be071d2e27cf32241c68bc70cc84e591259fc1a16eec47b"} Aug 01 00:33:58.334614 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:58.334122 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:33:58.350512 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:33:58.350446 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" podStartSLOduration=1.645939806 podStartE2EDuration="3.350423132s" totalImagesPullingTime="1.704483326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:58.350294018 +0000 UTC m=+287.707085676" watchObservedRunningTime="2026-08-01 00:33:58.350423132 +0000 UTC m=+287.707214783" Aug 01 00:34:00.345193 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:00.345144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" event={"ID":"db9fd642-bcbd-487e-95ef-5e3aba0cf39b","Type":"ContainerStarted","Data":"1e852893bd21bac2435c40b4138a24ad12d69269650fae6a2e3b514d3a82a75f"} Aug 01 00:34:00.361665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:00.361610 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6db59d85d8-rtr99" podStartSLOduration=2.382540019 podStartE2EDuration="5.361595281s" totalImagesPullingTime="2.979055262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:00.360024676 +0000 UTC m=+289.716816337" watchObservedRunningTime="2026-08-01 00:34:00.361595281 +0000 UTC m=+289.718386937" Aug 01 00:34:03.372628 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:03.372596 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:04.711796 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:04.711753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:05.584561 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:05.584521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:05.731929 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:05.731892 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:05.836379 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:05.836292 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:06.044993 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:06.044945 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:06.942568 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:06.942528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:07.050206 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:07.050146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:07.180632 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:07.180597 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:09.342196 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:09.342166 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dd57cb99d-9g67m" Aug 01 00:34:11.050579 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:11.050549 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:34:11.051186 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:11.051167 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:34:11.053273 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:11.053249 2575 kubelet.go:1648] "Image garbage collection succeeded" Aug 01 00:34:41.753928 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:41.753888 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:41.847418 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:41.847380 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:43.693685 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:43.693647 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:56.559476 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:56.559435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:56.658425 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:56.658386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:34:59.931554 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:59.931519 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl"] Aug 01 00:34:59.933884 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:59.933868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:34:59.936691 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:59.936670 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 01 00:34:59.937626 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:59.937614 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 01 00:34:59.937856 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:59.937826 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-844wz\"" Aug 01 00:34:59.943065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:34:59.943046 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl"] Aug 01 00:35:00.031663 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.031635 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f"] Aug 01 00:35:00.033903 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.033888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.042674 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.042651 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f"] Aug 01 00:35:00.072539 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.072510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.072539 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.072538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.072708 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.072564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56j7\" (UniqueName: \"kubernetes.io/projected/f0369b77-615b-440e-95a1-f5b1751dcce1-kube-api-access-l56j7\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.130091 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.130062 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq"] Aug 01 00:35:00.132837 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.132822 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.141239 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.141215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq"] Aug 01 00:35:00.173114 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.173247 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2mr\" (UniqueName: \"kubernetes.io/projected/722e7ee4-b46b-43e4-ac59-2a629b2faae8-kube-api-access-9c2mr\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.173247 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.173247 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.173410 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56j7\" (UniqueName: \"kubernetes.io/projected/f0369b77-615b-440e-95a1-f5b1751dcce1-kube-api-access-l56j7\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.173410 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.173481 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.173514 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.173492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.182212 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.182150 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56j7\" (UniqueName: \"kubernetes.io/projected/f0369b77-615b-440e-95a1-f5b1751dcce1-kube-api-access-l56j7\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.232545 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.232524 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf"] Aug 01 00:35:00.236473 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.236447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.243269 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.243244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf"] Aug 01 00:35:00.243380 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.243293 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:00.273841 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.273817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.273948 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.273873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.273948 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.273905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9ld\" (UniqueName: \"kubernetes.io/projected/5fad89e7-c92e-4afe-b054-e1e53091e197-kube-api-access-9g9ld\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.273948 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.273932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.274055 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.273985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.274055 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.274013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2mr\" (UniqueName: \"kubernetes.io/projected/722e7ee4-b46b-43e4-ac59-2a629b2faae8-kube-api-access-9c2mr\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.274222 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.274206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.274300 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.274280 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.286770 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.286742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2mr\" (UniqueName: \"kubernetes.io/projected/722e7ee4-b46b-43e4-ac59-2a629b2faae8-kube-api-access-9c2mr\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.343101 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.343074 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:00.370099 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.370076 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl"] Aug 01 00:35:00.372194 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:35:00.372168 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0369b77_615b_440e_95a1_f5b1751dcce1.slice/crio-14fc46b8f0123734442594db3aaf3b2e23e3df5539ac9b3846c080a37e90b93a WatchSource:0}: Error finding container 14fc46b8f0123734442594db3aaf3b2e23e3df5539ac9b3846c080a37e90b93a: Status 404 returned error can't find the container with id 14fc46b8f0123734442594db3aaf3b2e23e3df5539ac9b3846c080a37e90b93a Aug 01 00:35:00.374332 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.374437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.374437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.374560 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxxl\" (UniqueName: \"kubernetes.io/projected/3a9e48ab-b067-4845-8188-02d109ac48a9-kube-api-access-4cxxl\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.374675 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.374675 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9ld\" (UniqueName: \"kubernetes.io/projected/5fad89e7-c92e-4afe-b054-e1e53091e197-kube-api-access-9g9ld\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.374772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.374876 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.374856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.383121 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.383100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9ld\" (UniqueName: \"kubernetes.io/projected/5fad89e7-c92e-4afe-b054-e1e53091e197-kube-api-access-9g9ld\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.441954 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.441928 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:00.475584 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.475336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.475584 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.475390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxxl\" (UniqueName: \"kubernetes.io/projected/3a9e48ab-b067-4845-8188-02d109ac48a9-kube-api-access-4cxxl\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.475584 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.475463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.475879 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.475813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.475981 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.475954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.477480 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.477457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f"] Aug 01 00:35:00.479814 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:35:00.479790 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722e7ee4_b46b_43e4_ac59_2a629b2faae8.slice/crio-62fad4fe4808c86f49fe6935c64e83cedc54e669232dd8e5efd592a047e6b9bd WatchSource:0}: Error finding container 62fad4fe4808c86f49fe6935c64e83cedc54e669232dd8e5efd592a047e6b9bd: Status 404 returned error can't find the container with id 62fad4fe4808c86f49fe6935c64e83cedc54e669232dd8e5efd592a047e6b9bd Aug 01 00:35:00.484817 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.484795 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxxl\" (UniqueName: \"kubernetes.io/projected/3a9e48ab-b067-4845-8188-02d109ac48a9-kube-api-access-4cxxl\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.548135 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.548107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:00.569101 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.569079 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq"] Aug 01 00:35:00.571001 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:35:00.570972 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fad89e7_c92e_4afe_b054_e1e53091e197.slice/crio-48ad0440683dab13388ebb9720e28616e0eb6c73fecd3155fdd1b3b00324541c WatchSource:0}: Error finding container 48ad0440683dab13388ebb9720e28616e0eb6c73fecd3155fdd1b3b00324541c: Status 404 returned error can't find the container with id 48ad0440683dab13388ebb9720e28616e0eb6c73fecd3155fdd1b3b00324541c Aug 01 00:35:00.587889 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.587846 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" event={"ID":"5fad89e7-c92e-4afe-b054-e1e53091e197","Type":"ContainerStarted","Data":"48ad0440683dab13388ebb9720e28616e0eb6c73fecd3155fdd1b3b00324541c"} Aug 01 00:35:00.590888 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.590813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" event={"ID":"722e7ee4-b46b-43e4-ac59-2a629b2faae8","Type":"ContainerStarted","Data":"d7f553d085a5173fb0f455b28fdc6f29e81bff7d25b800802193aae140dae95b"} Aug 01 00:35:00.590888 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.590864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" event={"ID":"722e7ee4-b46b-43e4-ac59-2a629b2faae8","Type":"ContainerStarted","Data":"62fad4fe4808c86f49fe6935c64e83cedc54e669232dd8e5efd592a047e6b9bd"} Aug 01 00:35:00.592631 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.592609 2575 generic.go:356] "Generic (PLEG): container finished" podID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerID="9fea6900584c446843b17f48d8e7ed07ac1f161bae26671fd373fdc41b5d9cc7" exitCode=0 Aug 01 00:35:00.592707 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.592686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" event={"ID":"f0369b77-615b-440e-95a1-f5b1751dcce1","Type":"ContainerDied","Data":"9fea6900584c446843b17f48d8e7ed07ac1f161bae26671fd373fdc41b5d9cc7"} Aug 01 00:35:00.592755 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.592731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" event={"ID":"f0369b77-615b-440e-95a1-f5b1751dcce1","Type":"ContainerStarted","Data":"14fc46b8f0123734442594db3aaf3b2e23e3df5539ac9b3846c080a37e90b93a"} Aug 01 00:35:00.593623 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.593605 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:35:00.693591 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:00.693569 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf"] Aug 01 00:35:00.736991 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:35:00.736962 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9e48ab_b067_4845_8188_02d109ac48a9.slice/crio-a38e8a72bf69eed351aba0eceaf81857d959fca10c5785a7f9879ac44758f40c WatchSource:0}: Error finding container a38e8a72bf69eed351aba0eceaf81857d959fca10c5785a7f9879ac44758f40c: Status 404 returned error can't find the container with id a38e8a72bf69eed351aba0eceaf81857d959fca10c5785a7f9879ac44758f40c Aug 01 00:35:01.597552 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:01.597519 2575 generic.go:356] "Generic (PLEG): container finished" podID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerID="d7f553d085a5173fb0f455b28fdc6f29e81bff7d25b800802193aae140dae95b" exitCode=0 Aug 01 00:35:01.598023 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:01.597605 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" event={"ID":"722e7ee4-b46b-43e4-ac59-2a629b2faae8","Type":"ContainerDied","Data":"d7f553d085a5173fb0f455b28fdc6f29e81bff7d25b800802193aae140dae95b"} Aug 01 00:35:01.599245 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:01.599226 2575 generic.go:356] "Generic (PLEG): container finished" podID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerID="e46998744842c5aaf8063c3845c048d21c35d4fb69ac479b1177e30def44b4df" exitCode=0 Aug 01 00:35:01.599323 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:01.599302 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" event={"ID":"3a9e48ab-b067-4845-8188-02d109ac48a9","Type":"ContainerDied","Data":"e46998744842c5aaf8063c3845c048d21c35d4fb69ac479b1177e30def44b4df"} Aug 01 00:35:01.599389 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:01.599350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" event={"ID":"3a9e48ab-b067-4845-8188-02d109ac48a9","Type":"ContainerStarted","Data":"a38e8a72bf69eed351aba0eceaf81857d959fca10c5785a7f9879ac44758f40c"} Aug 01 00:35:01.600915 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:01.600897 2575 generic.go:356] "Generic (PLEG): container finished" podID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerID="ff8c2f02407a7cbba59c95b64bfcb223d5594b009953f2fcc38a538ee39fac9c" exitCode=0 Aug 01 00:35:01.600982 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:01.600922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" event={"ID":"5fad89e7-c92e-4afe-b054-e1e53091e197","Type":"ContainerDied","Data":"ff8c2f02407a7cbba59c95b64bfcb223d5594b009953f2fcc38a538ee39fac9c"} Aug 01 00:35:02.609080 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:02.609045 2575 generic.go:356] "Generic (PLEG): container finished" podID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerID="b35dd61edb1a362cc1f4ec15a4b7ceea7997e3d357199b49b5b02212e147b7bb" exitCode=0 Aug 01 00:35:02.609438 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:02.609117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" event={"ID":"f0369b77-615b-440e-95a1-f5b1751dcce1","Type":"ContainerDied","Data":"b35dd61edb1a362cc1f4ec15a4b7ceea7997e3d357199b49b5b02212e147b7bb"} Aug 01 00:35:03.617221 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.617185 2575 generic.go:356] "Generic (PLEG): container finished" podID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerID="99e7c42facbe533de0c36fd2ae7b84ef846358f3cf91aafa004dd082bd51ac72" exitCode=0 Aug 01 00:35:03.617615 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.617285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" event={"ID":"5fad89e7-c92e-4afe-b054-e1e53091e197","Type":"ContainerDied","Data":"99e7c42facbe533de0c36fd2ae7b84ef846358f3cf91aafa004dd082bd51ac72"} Aug 01 00:35:03.619097 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.619076 2575 generic.go:356] "Generic (PLEG): container finished" podID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerID="83e03499a3b946951d6d9202050ebeb55529568693c79669cfb0da9d5ab9d37b" exitCode=0 Aug 01 00:35:03.619212 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.619099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" event={"ID":"722e7ee4-b46b-43e4-ac59-2a629b2faae8","Type":"ContainerDied","Data":"83e03499a3b946951d6d9202050ebeb55529568693c79669cfb0da9d5ab9d37b"} Aug 01 00:35:03.621148 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.621128 2575 generic.go:356] "Generic (PLEG): container finished" podID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerID="b433999779ce9a0e2b8b57b3b6981c23aa6d295ee26bfbac6f8ee496080c80cc" exitCode=0 Aug 01 00:35:03.621232 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.621191 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" event={"ID":"f0369b77-615b-440e-95a1-f5b1751dcce1","Type":"ContainerDied","Data":"b433999779ce9a0e2b8b57b3b6981c23aa6d295ee26bfbac6f8ee496080c80cc"} Aug 01 00:35:03.622920 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.622898 2575 generic.go:356] "Generic (PLEG): container finished" podID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerID="91f80147797590902d46083a300159bc984282dc5e138f9c191865ecb0a4e261" exitCode=0 Aug 01 00:35:03.623043 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:03.622961 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" event={"ID":"3a9e48ab-b067-4845-8188-02d109ac48a9","Type":"ContainerDied","Data":"91f80147797590902d46083a300159bc984282dc5e138f9c191865ecb0a4e261"} Aug 01 00:35:04.629581 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.629548 2575 generic.go:356] "Generic (PLEG): container finished" podID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerID="ff66951e3317e941ff35a7e9cb3cb9ed7d0657a7beb65fa82d8e211c9a6a99eb" exitCode=0 Aug 01 00:35:04.630010 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.629637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" event={"ID":"5fad89e7-c92e-4afe-b054-e1e53091e197","Type":"ContainerDied","Data":"ff66951e3317e941ff35a7e9cb3cb9ed7d0657a7beb65fa82d8e211c9a6a99eb"} Aug 01 00:35:04.631523 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.631500 2575 generic.go:356] "Generic (PLEG): container finished" podID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerID="f90d620a48b09af7071aca8804e6e1ee6b23196f01225f6dfa56a118f12c03cb" exitCode=0 Aug 01 00:35:04.631630 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.631576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" event={"ID":"722e7ee4-b46b-43e4-ac59-2a629b2faae8","Type":"ContainerDied","Data":"f90d620a48b09af7071aca8804e6e1ee6b23196f01225f6dfa56a118f12c03cb"} Aug 01 00:35:04.633511 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.633489 2575 generic.go:356] "Generic (PLEG): container finished" podID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerID="eab5180c7ac75b7a0ac80da4d9cba6427a55b20eb7879a5d7fb810a981993e80" exitCode=0 Aug 01 00:35:04.633620 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.633568 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" event={"ID":"3a9e48ab-b067-4845-8188-02d109ac48a9","Type":"ContainerDied","Data":"eab5180c7ac75b7a0ac80da4d9cba6427a55b20eb7879a5d7fb810a981993e80"} Aug 01 00:35:04.761924 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.761899 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:04.906459 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.906417 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-util\") pod \"f0369b77-615b-440e-95a1-f5b1751dcce1\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " Aug 01 00:35:04.906459 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.906475 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-bundle\") pod \"f0369b77-615b-440e-95a1-f5b1751dcce1\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " Aug 01 00:35:04.906670 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.906509 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l56j7\" (UniqueName: \"kubernetes.io/projected/f0369b77-615b-440e-95a1-f5b1751dcce1-kube-api-access-l56j7\") pod \"f0369b77-615b-440e-95a1-f5b1751dcce1\" (UID: \"f0369b77-615b-440e-95a1-f5b1751dcce1\") " Aug 01 00:35:04.906984 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.906961 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-bundle" (OuterVolumeSpecName: "bundle") pod "f0369b77-615b-440e-95a1-f5b1751dcce1" (UID: "f0369b77-615b-440e-95a1-f5b1751dcce1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:04.908628 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.908600 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0369b77-615b-440e-95a1-f5b1751dcce1-kube-api-access-l56j7" (OuterVolumeSpecName: "kube-api-access-l56j7") pod "f0369b77-615b-440e-95a1-f5b1751dcce1" (UID: "f0369b77-615b-440e-95a1-f5b1751dcce1"). InnerVolumeSpecName "kube-api-access-l56j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:35:04.912166 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:04.912138 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-util" (OuterVolumeSpecName: "util") pod "f0369b77-615b-440e-95a1-f5b1751dcce1" (UID: "f0369b77-615b-440e-95a1-f5b1751dcce1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:05.007271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.007198 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:05.007271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.007224 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0369b77-615b-440e-95a1-f5b1751dcce1-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:05.007271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.007233 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l56j7\" (UniqueName: \"kubernetes.io/projected/f0369b77-615b-440e-95a1-f5b1751dcce1-kube-api-access-l56j7\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:05.639652 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.639626 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" Aug 01 00:35:05.640084 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.639675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ctx8fl" event={"ID":"f0369b77-615b-440e-95a1-f5b1751dcce1","Type":"ContainerDied","Data":"14fc46b8f0123734442594db3aaf3b2e23e3df5539ac9b3846c080a37e90b93a"} Aug 01 00:35:05.640084 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.639718 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14fc46b8f0123734442594db3aaf3b2e23e3df5539ac9b3846c080a37e90b93a" Aug 01 00:35:05.781496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.781474 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:05.816538 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.816485 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:05.820298 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.820280 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:05.913325 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-util\") pod \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " Aug 01 00:35:05.913325 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-bundle\") pod \"3a9e48ab-b067-4845-8188-02d109ac48a9\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " Aug 01 00:35:05.913527 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-util\") pod \"3a9e48ab-b067-4845-8188-02d109ac48a9\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " Aug 01 00:35:05.913527 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913364 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-util\") pod \"5fad89e7-c92e-4afe-b054-e1e53091e197\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " Aug 01 00:35:05.913527 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913384 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cxxl\" (UniqueName: \"kubernetes.io/projected/3a9e48ab-b067-4845-8188-02d109ac48a9-kube-api-access-4cxxl\") pod \"3a9e48ab-b067-4845-8188-02d109ac48a9\" (UID: \"3a9e48ab-b067-4845-8188-02d109ac48a9\") " Aug 01 00:35:05.913527 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913441 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-bundle\") pod \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " Aug 01 00:35:05.913527 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913479 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-bundle\") pod \"5fad89e7-c92e-4afe-b054-e1e53091e197\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " Aug 01 00:35:05.913527 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913514 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g9ld\" (UniqueName: \"kubernetes.io/projected/5fad89e7-c92e-4afe-b054-e1e53091e197-kube-api-access-9g9ld\") pod \"5fad89e7-c92e-4afe-b054-e1e53091e197\" (UID: \"5fad89e7-c92e-4afe-b054-e1e53091e197\") " Aug 01 00:35:05.913922 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.913555 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c2mr\" (UniqueName: \"kubernetes.io/projected/722e7ee4-b46b-43e4-ac59-2a629b2faae8-kube-api-access-9c2mr\") pod \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\" (UID: \"722e7ee4-b46b-43e4-ac59-2a629b2faae8\") " Aug 01 00:35:05.914374 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.914066 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-bundle" (OuterVolumeSpecName: "bundle") pod "3a9e48ab-b067-4845-8188-02d109ac48a9" (UID: "3a9e48ab-b067-4845-8188-02d109ac48a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:05.914479 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.914394 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-bundle" (OuterVolumeSpecName: "bundle") pod "5fad89e7-c92e-4afe-b054-e1e53091e197" (UID: "5fad89e7-c92e-4afe-b054-e1e53091e197"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:05.914536 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.914497 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-bundle" (OuterVolumeSpecName: "bundle") pod "722e7ee4-b46b-43e4-ac59-2a629b2faae8" (UID: "722e7ee4-b46b-43e4-ac59-2a629b2faae8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:05.916245 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.916148 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9e48ab-b067-4845-8188-02d109ac48a9-kube-api-access-4cxxl" (OuterVolumeSpecName: "kube-api-access-4cxxl") pod "3a9e48ab-b067-4845-8188-02d109ac48a9" (UID: "3a9e48ab-b067-4845-8188-02d109ac48a9"). InnerVolumeSpecName "kube-api-access-4cxxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:35:05.916358 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.916250 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722e7ee4-b46b-43e4-ac59-2a629b2faae8-kube-api-access-9c2mr" (OuterVolumeSpecName: "kube-api-access-9c2mr") pod "722e7ee4-b46b-43e4-ac59-2a629b2faae8" (UID: "722e7ee4-b46b-43e4-ac59-2a629b2faae8"). InnerVolumeSpecName "kube-api-access-9c2mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:35:05.916404 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.916380 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fad89e7-c92e-4afe-b054-e1e53091e197-kube-api-access-9g9ld" (OuterVolumeSpecName: "kube-api-access-9g9ld") pod "5fad89e7-c92e-4afe-b054-e1e53091e197" (UID: "5fad89e7-c92e-4afe-b054-e1e53091e197"). InnerVolumeSpecName "kube-api-access-9g9ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:35:05.919883 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.919848 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-util" (OuterVolumeSpecName: "util") pod "5fad89e7-c92e-4afe-b054-e1e53091e197" (UID: "5fad89e7-c92e-4afe-b054-e1e53091e197"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:05.919989 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.919971 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-util" (OuterVolumeSpecName: "util") pod "3a9e48ab-b067-4845-8188-02d109ac48a9" (UID: "3a9e48ab-b067-4845-8188-02d109ac48a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:05.920537 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:05.920516 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-util" (OuterVolumeSpecName: "util") pod "722e7ee4-b46b-43e4-ac59-2a629b2faae8" (UID: "722e7ee4-b46b-43e4-ac59-2a629b2faae8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:35:06.014744 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014694 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9ld\" (UniqueName: \"kubernetes.io/projected/5fad89e7-c92e-4afe-b054-e1e53091e197-kube-api-access-9g9ld\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.014744 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014736 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2mr\" (UniqueName: \"kubernetes.io/projected/722e7ee4-b46b-43e4-ac59-2a629b2faae8-kube-api-access-9c2mr\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.014744 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014745 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.014744 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014755 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.014744 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014762 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a9e48ab-b067-4845-8188-02d109ac48a9-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.015030 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014770 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-util\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.015030 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014777 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cxxl\" (UniqueName: \"kubernetes.io/projected/3a9e48ab-b067-4845-8188-02d109ac48a9-kube-api-access-4cxxl\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.015030 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014786 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/722e7ee4-b46b-43e4-ac59-2a629b2faae8-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.015030 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.014793 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fad89e7-c92e-4afe-b054-e1e53091e197-bundle\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:35:06.645910 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.645877 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" Aug 01 00:35:06.645910 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.645890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053qjtlq" event={"ID":"5fad89e7-c92e-4afe-b054-e1e53091e197","Type":"ContainerDied","Data":"48ad0440683dab13388ebb9720e28616e0eb6c73fecd3155fdd1b3b00324541c"} Aug 01 00:35:06.646383 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.645937 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ad0440683dab13388ebb9720e28616e0eb6c73fecd3155fdd1b3b00324541c" Aug 01 00:35:06.647698 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.647675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" event={"ID":"722e7ee4-b46b-43e4-ac59-2a629b2faae8","Type":"ContainerDied","Data":"62fad4fe4808c86f49fe6935c64e83cedc54e669232dd8e5efd592a047e6b9bd"} Aug 01 00:35:06.647798 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.647714 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62fad4fe4808c86f49fe6935c64e83cedc54e669232dd8e5efd592a047e6b9bd" Aug 01 00:35:06.647798 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.647685 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0tkt2f" Aug 01 00:35:06.649518 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.649499 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" Aug 01 00:35:06.649617 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.649504 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbbkhwf" event={"ID":"3a9e48ab-b067-4845-8188-02d109ac48a9","Type":"ContainerDied","Data":"a38e8a72bf69eed351aba0eceaf81857d959fca10c5785a7f9879ac44758f40c"} Aug 01 00:35:06.649617 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:06.649601 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a38e8a72bf69eed351aba0eceaf81857d959fca10c5785a7f9879ac44758f40c" Aug 01 00:35:14.273144 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:14.273106 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:17.880629 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:17.880581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:18.508548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:18.508519 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:19.402319 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:19.402284 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:20.941617 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:20.941582 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:22.394454 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:22.394421 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:22.667267 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:22.667238 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:22.903616 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:22.903582 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:23.270045 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:23.270011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:25.759100 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:25.759057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:25.848933 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:25.848894 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:28.661211 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:28.661171 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:29.505985 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:29.505947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:30.121419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:30.121388 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:30.222558 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:30.222523 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:30.321401 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:30.321367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:30.419087 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:30.419059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:33.271438 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:33.271396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:33.368776 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:33.368736 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:36.248173 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248130 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq"] Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248416 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248428 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248439 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248444 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248453 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248459 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248464 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerName="pull" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248470 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerName="pull" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248479 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248485 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248490 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248495 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248502 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248506 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248515 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248519 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerName="extract" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248527 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerName="pull" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248531 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerName="pull" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248539 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerName="pull" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248544 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerName="pull" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248551 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248556 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerName="util" Aug 01 00:35:36.248551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248561 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerName="pull" Aug 01 00:35:36.249233 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248566 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerName="pull" Aug 01 00:35:36.249233 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248608 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0369b77-615b-440e-95a1-f5b1751dcce1" containerName="extract" Aug 01 00:35:36.249233 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248616 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9e48ab-b067-4845-8188-02d109ac48a9" containerName="extract" Aug 01 00:35:36.249233 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248621 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fad89e7-c92e-4afe-b054-e1e53091e197" containerName="extract" Aug 01 00:35:36.249233 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.248626 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="722e7ee4-b46b-43e4-ac59-2a629b2faae8" containerName="extract" Aug 01 00:35:36.251341 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.251326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.253764 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.253696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 01 00:35:36.253764 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.253717 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 01 00:35:36.253979 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.253781 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 01 00:35:36.254672 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.254656 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zrp4w\"" Aug 01 00:35:36.254732 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.254656 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 01 00:35:36.261395 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.261366 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:36.262066 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.262046 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq"] Aug 01 00:35:36.361391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.361350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0021dd34-6670-459b-8210-889f1546e340-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.361391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.361392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0021dd34-6670-459b-8210-889f1546e340-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.361596 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.361470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2b8s\" (UniqueName: \"kubernetes.io/projected/0021dd34-6670-459b-8210-889f1546e340-kube-api-access-t2b8s\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.365141 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.365111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:36.461871 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.461825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2b8s\" (UniqueName: \"kubernetes.io/projected/0021dd34-6670-459b-8210-889f1546e340-kube-api-access-t2b8s\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.462070 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.461912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0021dd34-6670-459b-8210-889f1546e340-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.462070 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.461937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0021dd34-6670-459b-8210-889f1546e340-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.462659 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.462634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0021dd34-6670-459b-8210-889f1546e340-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.464303 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.464272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0021dd34-6670-459b-8210-889f1546e340-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.470226 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.470201 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2b8s\" (UniqueName: \"kubernetes.io/projected/0021dd34-6670-459b-8210-889f1546e340-kube-api-access-t2b8s\") pod \"kuadrant-console-plugin-598f86468b-8dplq\" (UID: \"0021dd34-6670-459b-8210-889f1546e340\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.561271 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.561180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" Aug 01 00:35:36.697664 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.697641 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq"] Aug 01 00:35:36.699949 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:35:36.699915 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0021dd34_6670_459b_8210_889f1546e340.slice/crio-13d3d688a836df57a12d3ebdb61f02258dc183ce176bdea428e874635fca2709 WatchSource:0}: Error finding container 13d3d688a836df57a12d3ebdb61f02258dc183ce176bdea428e874635fca2709: Status 404 returned error can't find the container with id 13d3d688a836df57a12d3ebdb61f02258dc183ce176bdea428e874635fca2709 Aug 01 00:35:36.781694 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:36.781657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" event={"ID":"0021dd34-6670-459b-8210-889f1546e340","Type":"ContainerStarted","Data":"13d3d688a836df57a12d3ebdb61f02258dc183ce176bdea428e874635fca2709"} Aug 01 00:35:38.461502 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:38.461464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:38.562733 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:38.562642 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:39.463532 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:39.463496 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:39.522173 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:39.522120 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:42.815123 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:42.815086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" event={"ID":"0021dd34-6670-459b-8210-889f1546e340","Type":"ContainerStarted","Data":"bda7c27c67dfaec3730cb78e6bb8c125ee1d4429c4f4194e9866170637450622"} Aug 01 00:35:42.830842 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:42.830792 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-8dplq" podStartSLOduration=1.710214144 podStartE2EDuration="6.830776848s" totalImagesPullingTime="5.120562704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:42.829276568 +0000 UTC m=+392.186068224" watchObservedRunningTime="2026-08-01 00:35:42.830776848 +0000 UTC m=+392.187568505" Aug 01 00:35:45.712768 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:45.712733 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:45.814574 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:45.814540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:46.071916 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:46.071832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:46.170895 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:46.170859 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:46.838130 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:46.838092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:46.939904 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:46.939867 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:53.664722 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:53.664679 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:53.764797 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:53.764761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:58.280471 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:58.280433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:35:58.374923 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:35:58.374879 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:00.854670 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:00.854632 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:00.955617 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:00.955577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:03.466209 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:03.466179 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:03.517781 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:03.517743 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:04.260918 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:04.260882 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:04.319989 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:04.319954 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:10.660457 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:10.660419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:10.719106 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:10.719067 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:11.268809 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:11.268776 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:11.367660 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:11.367607 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:11.865918 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:11.865878 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:11.966917 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:11.966877 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:18.663464 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:18.663429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:18.764499 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:18.764467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:26.019349 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:26.019312 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:26.944935 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:26.944900 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:27.045898 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:27.045864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:30.982441 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:30.982384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:31.385535 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:31.385442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:33.234172 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.234062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:33.345317 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.345281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:33.401628 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.401590 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:33.638396 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.638312 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:36:33.641463 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.641446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:33.643952 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.643927 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 01 00:36:33.651985 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.651959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:36:33.685919 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.685878 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:36:33.710536 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.710501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c93cba19-7d5d-42a6-8f31-409aa20e1c54-config-file\") pod \"limitador-limitador-85d9db4d5d-nh9jc\" (UID: \"c93cba19-7d5d-42a6-8f31-409aa20e1c54\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:33.710705 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.710557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgrw\" (UniqueName: \"kubernetes.io/projected/c93cba19-7d5d-42a6-8f31-409aa20e1c54-kube-api-access-bfgrw\") pod \"limitador-limitador-85d9db4d5d-nh9jc\" (UID: \"c93cba19-7d5d-42a6-8f31-409aa20e1c54\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:33.811731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.811695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgrw\" (UniqueName: \"kubernetes.io/projected/c93cba19-7d5d-42a6-8f31-409aa20e1c54-kube-api-access-bfgrw\") pod \"limitador-limitador-85d9db4d5d-nh9jc\" (UID: \"c93cba19-7d5d-42a6-8f31-409aa20e1c54\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:33.811941 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.811793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c93cba19-7d5d-42a6-8f31-409aa20e1c54-config-file\") pod \"limitador-limitador-85d9db4d5d-nh9jc\" (UID: \"c93cba19-7d5d-42a6-8f31-409aa20e1c54\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:33.812463 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.812442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c93cba19-7d5d-42a6-8f31-409aa20e1c54-config-file\") pod \"limitador-limitador-85d9db4d5d-nh9jc\" (UID: \"c93cba19-7d5d-42a6-8f31-409aa20e1c54\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:33.819849 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.819825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgrw\" (UniqueName: \"kubernetes.io/projected/c93cba19-7d5d-42a6-8f31-409aa20e1c54-kube-api-access-bfgrw\") pod \"limitador-limitador-85d9db4d5d-nh9jc\" (UID: \"c93cba19-7d5d-42a6-8f31-409aa20e1c54\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:33.951587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:33.951550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:34.084025 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.083994 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:34.089376 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.089351 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:36:34.091941 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:36:34.091910 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93cba19_7d5d_42a6_8f31_409aa20e1c54.slice/crio-82315af08f58976ff46ded0abe00266665383932f554ef0a82a4b7248804f7bb WatchSource:0}: Error finding container 82315af08f58976ff46ded0abe00266665383932f554ef0a82a4b7248804f7bb: Status 404 returned error can't find the container with id 82315af08f58976ff46ded0abe00266665383932f554ef0a82a4b7248804f7bb Aug 01 00:36:34.186058 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.186026 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:34.194911 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.194882 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-74sh7"] Aug 01 00:36:34.199563 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.199543 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-74sh7" Aug 01 00:36:34.202353 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.202288 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vpbxr\"" Aug 01 00:36:34.202917 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.202898 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-74sh7"] Aug 01 00:36:34.215543 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.215511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2dn\" (UniqueName: \"kubernetes.io/projected/60e32803-6e15-4f8e-8eb8-a864df44bb49-kube-api-access-8j2dn\") pod \"authorino-c548b5d7d-74sh7\" (UID: \"60e32803-6e15-4f8e-8eb8-a864df44bb49\") " pod="kuadrant-system/authorino-c548b5d7d-74sh7" Aug 01 00:36:34.316229 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.316194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2dn\" (UniqueName: \"kubernetes.io/projected/60e32803-6e15-4f8e-8eb8-a864df44bb49-kube-api-access-8j2dn\") pod \"authorino-c548b5d7d-74sh7\" (UID: \"60e32803-6e15-4f8e-8eb8-a864df44bb49\") " pod="kuadrant-system/authorino-c548b5d7d-74sh7" Aug 01 00:36:34.323694 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.323665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2dn\" (UniqueName: \"kubernetes.io/projected/60e32803-6e15-4f8e-8eb8-a864df44bb49-kube-api-access-8j2dn\") pod \"authorino-c548b5d7d-74sh7\" (UID: \"60e32803-6e15-4f8e-8eb8-a864df44bb49\") " pod="kuadrant-system/authorino-c548b5d7d-74sh7" Aug 01 00:36:34.509505 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.509412 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-74sh7" Aug 01 00:36:34.637138 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:34.637114 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-74sh7"] Aug 01 00:36:34.639451 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:36:34.639426 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e32803_6e15_4f8e_8eb8_a864df44bb49.slice/crio-4b175b26b2006a0f89b57b23fdb0590882c190d1e4703672abd3047c194431df WatchSource:0}: Error finding container 4b175b26b2006a0f89b57b23fdb0590882c190d1e4703672abd3047c194431df: Status 404 returned error can't find the container with id 4b175b26b2006a0f89b57b23fdb0590882c190d1e4703672abd3047c194431df Aug 01 00:36:35.050587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:35.050546 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-74sh7" event={"ID":"60e32803-6e15-4f8e-8eb8-a864df44bb49","Type":"ContainerStarted","Data":"4b175b26b2006a0f89b57b23fdb0590882c190d1e4703672abd3047c194431df"} Aug 01 00:36:35.051679 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:35.051653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" event={"ID":"c93cba19-7d5d-42a6-8f31-409aa20e1c54","Type":"ContainerStarted","Data":"82315af08f58976ff46ded0abe00266665383932f554ef0a82a4b7248804f7bb"} Aug 01 00:36:39.074463 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:39.074421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" event={"ID":"c93cba19-7d5d-42a6-8f31-409aa20e1c54","Type":"ContainerStarted","Data":"13b67af6e2f30ffbeec52eedb47a02f42f6aecd04daf2339d29754e4ca697018"} Aug 01 00:36:39.074921 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:39.074495 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:39.075880 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:39.075851 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-74sh7" event={"ID":"60e32803-6e15-4f8e-8eb8-a864df44bb49","Type":"ContainerStarted","Data":"65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76"} Aug 01 00:36:39.092516 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:39.092465 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" podStartSLOduration=2.105388027 podStartE2EDuration="6.092447568s" totalImagesPullingTime="3.987059541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:36:39.089768841 +0000 UTC m=+448.446560496" watchObservedRunningTime="2026-08-01 00:36:39.092447568 +0000 UTC m=+448.449239223" Aug 01 00:36:39.103127 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:39.103076 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-74sh7" podStartSLOduration=1.656801389 podStartE2EDuration="5.10305947s" totalImagesPullingTime="3.446258081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:36:39.102776003 +0000 UTC m=+448.459567659" watchObservedRunningTime="2026-08-01 00:36:39.10305947 +0000 UTC m=+448.459851126" Aug 01 00:36:50.082422 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:50.082389 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc" Aug 01 00:36:54.456223 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:54.456185 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:54.558505 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:54.558472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:36:54.919501 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:36:54.919468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:08.817319 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:08.817282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:09.031583 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.031549 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rthmt"] Aug 01 00:37:09.034685 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.034668 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" Aug 01 00:37:09.041280 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.041252 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rthmt"] Aug 01 00:37:09.058544 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.058501 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:09.062520 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.062500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxsr9\" (UniqueName: \"kubernetes.io/projected/65112a9b-6058-43df-839e-c791ddaf60f4-kube-api-access-sxsr9\") pod \"authorino-5dc7ccb5f-rthmt\" (UID: \"65112a9b-6058-43df-839e-c791ddaf60f4\") " pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" Aug 01 00:37:09.163563 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.163530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxsr9\" (UniqueName: \"kubernetes.io/projected/65112a9b-6058-43df-839e-c791ddaf60f4-kube-api-access-sxsr9\") pod \"authorino-5dc7ccb5f-rthmt\" (UID: \"65112a9b-6058-43df-839e-c791ddaf60f4\") " pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" Aug 01 00:37:09.164123 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.164106 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rthmt"] Aug 01 00:37:09.164321 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:37:09.164304 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sxsr9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" podUID="65112a9b-6058-43df-839e-c791ddaf60f4" Aug 01 00:37:09.171857 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.171834 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxsr9\" (UniqueName: \"kubernetes.io/projected/65112a9b-6058-43df-839e-c791ddaf60f4-kube-api-access-sxsr9\") pod \"authorino-5dc7ccb5f-rthmt\" (UID: \"65112a9b-6058-43df-839e-c791ddaf60f4\") " pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" Aug 01 00:37:09.187485 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.187452 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-2qcj4"] Aug 01 00:37:09.190581 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.190564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:09.193320 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.193299 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 01 00:37:09.200661 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.200637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-2qcj4"] Aug 01 00:37:09.216659 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.216628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" Aug 01 00:37:09.223710 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.223685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" Aug 01 00:37:09.262494 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.262456 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-2qcj4"] Aug 01 00:37:09.262969 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:37:09.262943 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7gcbf tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-2qcj4" podUID="d094c3c8-4bdb-4d6e-b210-8a34cfaa7051" Aug 01 00:37:09.265124 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.264372 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxsr9\" (UniqueName: \"kubernetes.io/projected/65112a9b-6058-43df-839e-c791ddaf60f4-kube-api-access-sxsr9\") pod \"65112a9b-6058-43df-839e-c791ddaf60f4\" (UID: \"65112a9b-6058-43df-839e-c791ddaf60f4\") " Aug 01 00:37:09.265124 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.264617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-tls-cert\") pod \"authorino-687bd45585-2qcj4\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:09.265124 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.264677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcbf\" (UniqueName: \"kubernetes.io/projected/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-kube-api-access-7gcbf\") pod \"authorino-687bd45585-2qcj4\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:09.267686 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.267653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65112a9b-6058-43df-839e-c791ddaf60f4-kube-api-access-sxsr9" (OuterVolumeSpecName: "kube-api-access-sxsr9") pod "65112a9b-6058-43df-839e-c791ddaf60f4" (UID: "65112a9b-6058-43df-839e-c791ddaf60f4"). InnerVolumeSpecName "kube-api-access-sxsr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:37:09.365926 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.365884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-tls-cert\") pod \"authorino-687bd45585-2qcj4\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:09.366118 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.365953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcbf\" (UniqueName: \"kubernetes.io/projected/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-kube-api-access-7gcbf\") pod \"authorino-687bd45585-2qcj4\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:09.366118 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.366011 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxsr9\" (UniqueName: \"kubernetes.io/projected/65112a9b-6058-43df-839e-c791ddaf60f4-kube-api-access-sxsr9\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:37:09.368329 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.368309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-tls-cert\") pod \"authorino-687bd45585-2qcj4\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:09.373088 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:09.373062 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcbf\" (UniqueName: \"kubernetes.io/projected/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-kube-api-access-7gcbf\") pod \"authorino-687bd45585-2qcj4\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:10.220824 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.220789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:10.221301 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.220798 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rthmt" Aug 01 00:37:10.230823 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.230800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:10.268138 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.268103 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rthmt"] Aug 01 00:37:10.270302 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.270278 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rthmt"] Aug 01 00:37:10.274257 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.274232 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gcbf\" (UniqueName: \"kubernetes.io/projected/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-kube-api-access-7gcbf\") pod \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " Aug 01 00:37:10.274387 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.274281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-tls-cert\") pod \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\" (UID: \"d094c3c8-4bdb-4d6e-b210-8a34cfaa7051\") " Aug 01 00:37:10.276384 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.276352 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d094c3c8-4bdb-4d6e-b210-8a34cfaa7051" (UID: "d094c3c8-4bdb-4d6e-b210-8a34cfaa7051"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:37:10.276479 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.276364 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-kube-api-access-7gcbf" (OuterVolumeSpecName: "kube-api-access-7gcbf") pod "d094c3c8-4bdb-4d6e-b210-8a34cfaa7051" (UID: "d094c3c8-4bdb-4d6e-b210-8a34cfaa7051"). InnerVolumeSpecName "kube-api-access-7gcbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:37:10.375082 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.375041 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcbf\" (UniqueName: \"kubernetes.io/projected/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-kube-api-access-7gcbf\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:37:10.375082 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.375072 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051-tls-cert\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:37:10.562329 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.562253 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-74sh7"] Aug 01 00:37:10.562462 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.562417 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-74sh7" podUID="60e32803-6e15-4f8e-8eb8-a864df44bb49" containerName="authorino" containerID="cri-o://65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76" gracePeriod=30 Aug 01 00:37:10.835782 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.835701 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-74sh7" Aug 01 00:37:10.880621 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.880591 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j2dn\" (UniqueName: \"kubernetes.io/projected/60e32803-6e15-4f8e-8eb8-a864df44bb49-kube-api-access-8j2dn\") pod \"60e32803-6e15-4f8e-8eb8-a864df44bb49\" (UID: \"60e32803-6e15-4f8e-8eb8-a864df44bb49\") " Aug 01 00:37:10.882775 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.882747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e32803-6e15-4f8e-8eb8-a864df44bb49-kube-api-access-8j2dn" (OuterVolumeSpecName: "kube-api-access-8j2dn") pod "60e32803-6e15-4f8e-8eb8-a864df44bb49" (UID: "60e32803-6e15-4f8e-8eb8-a864df44bb49"). InnerVolumeSpecName "kube-api-access-8j2dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:37:10.981434 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:10.981403 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j2dn\" (UniqueName: \"kubernetes.io/projected/60e32803-6e15-4f8e-8eb8-a864df44bb49-kube-api-access-8j2dn\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:37:11.163105 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.163068 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65112a9b-6058-43df-839e-c791ddaf60f4" path="/var/lib/kubelet/pods/65112a9b-6058-43df-839e-c791ddaf60f4/volumes" Aug 01 00:37:11.226944 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.226909 2575 generic.go:356] "Generic (PLEG): container finished" podID="60e32803-6e15-4f8e-8eb8-a864df44bb49" containerID="65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76" exitCode=0 Aug 01 00:37:11.227437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.226957 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-74sh7" Aug 01 00:37:11.227437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.226997 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-74sh7" event={"ID":"60e32803-6e15-4f8e-8eb8-a864df44bb49","Type":"ContainerDied","Data":"65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76"} Aug 01 00:37:11.227437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.227056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-74sh7" event={"ID":"60e32803-6e15-4f8e-8eb8-a864df44bb49","Type":"ContainerDied","Data":"4b175b26b2006a0f89b57b23fdb0590882c190d1e4703672abd3047c194431df"} Aug 01 00:37:11.227437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.227099 2575 scope.go:117] "RemoveContainer" containerID="65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76" Aug 01 00:37:11.227437 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.227188 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2qcj4" Aug 01 00:37:11.235958 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.235940 2575 scope.go:117] "RemoveContainer" containerID="65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76" Aug 01 00:37:11.236363 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:37:11.236334 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76\": container with ID starting with 65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76 not found: ID does not exist" containerID="65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76" Aug 01 00:37:11.236427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.236366 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76"} err="failed to get container status \"65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76\": rpc error: code = NotFound desc = could not find container \"65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76\": container with ID starting with 65c00642b386ed1b3a25086dde8c07abad47a9e56926c3a8fb9e5a9ad8025b76 not found: ID does not exist" Aug 01 00:37:11.264682 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.264644 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-2qcj4"] Aug 01 00:37:11.270704 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.270676 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-2qcj4"] Aug 01 00:37:11.280943 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.280907 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-74sh7"] Aug 01 00:37:11.282747 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:11.282727 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-74sh7"] Aug 01 00:37:13.162742 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:13.162707 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e32803-6e15-4f8e-8eb8-a864df44bb49" path="/var/lib/kubelet/pods/60e32803-6e15-4f8e-8eb8-a864df44bb49/volumes" Aug 01 00:37:13.163089 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:13.163043 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d094c3c8-4bdb-4d6e-b210-8a34cfaa7051" path="/var/lib/kubelet/pods/d094c3c8-4bdb-4d6e-b210-8a34cfaa7051/volumes" Aug 01 00:37:14.945690 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:14.945647 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:15.043470 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:15.043438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:15.116312 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:15.116276 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:17.008982 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:17.008942 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:17.112973 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:17.112934 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:48.615841 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:48.615804 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:48.718488 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:48.718453 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:50.182109 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:50.182054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:50.621122 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:50.621041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:50.726185 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:50.726131 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:50.775729 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:50.775676 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:37:50.807388 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:50.807347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:52.314017 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:52.313983 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:37:52.366367 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:37:52.366337 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:38:00.693335 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:00.693295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:38:00.792073 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:00.792033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:38:09.571304 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:09.571271 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:38:09.967509 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:09.967478 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:38:24.971106 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:24.971075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:38:25.567956 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:25.567923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:38:42.870540 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:42.870493 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:38:43.763496 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:43.763458 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:38:57.656453 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:38:57.656417 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:39:02.445952 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.445920 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft"] Aug 01 00:39:02.446352 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.446231 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e32803-6e15-4f8e-8eb8-a864df44bb49" containerName="authorino" Aug 01 00:39:02.446352 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.446243 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e32803-6e15-4f8e-8eb8-a864df44bb49" containerName="authorino" Aug 01 00:39:02.446352 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.446298 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e32803-6e15-4f8e-8eb8-a864df44bb49" containerName="authorino" Aug 01 00:39:02.449339 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.449323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.451850 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.451828 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 01 00:39:02.452710 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.452688 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 01 00:39:02.452810 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.452717 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fzrd7\"" Aug 01 00:39:02.452862 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.452827 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 01 00:39:02.461071 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.461045 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft"] Aug 01 00:39:02.486320 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.486285 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:39:02.524373 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.524336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbe12152-f594-4f89-85a0-26f490f11c97-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.524373 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.524379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.524626 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.524397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.524626 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.524485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.524626 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.524565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.524755 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.524623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55k6n\" (UniqueName: \"kubernetes.io/projected/bbe12152-f594-4f89-85a0-26f490f11c97-kube-api-access-55k6n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625404 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625620 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55k6n\" (UniqueName: \"kubernetes.io/projected/bbe12152-f594-4f89-85a0-26f490f11c97-kube-api-access-55k6n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625620 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbe12152-f594-4f89-85a0-26f490f11c97-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625620 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625620 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625620 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625896 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625938 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.625938 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.625916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.627809 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.627788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbe12152-f594-4f89-85a0-26f490f11c97-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.627966 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.627948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbe12152-f594-4f89-85a0-26f490f11c97-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.633629 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.633602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55k6n\" (UniqueName: \"kubernetes.io/projected/bbe12152-f594-4f89-85a0-26f490f11c97-kube-api-access-55k6n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft\" (UID: \"bbe12152-f594-4f89-85a0-26f490f11c97\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.759659 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.759550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:02.894949 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:02.894906 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft"] Aug 01 00:39:02.897942 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:39:02.897914 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe12152_f594_4f89_85a0_26f490f11c97.slice/crio-99e38e60a9b6043d3b782294240b61b4bfbd8f8811a9070f0a4865a7a9fe2cfa WatchSource:0}: Error finding container 99e38e60a9b6043d3b782294240b61b4bfbd8f8811a9070f0a4865a7a9fe2cfa: Status 404 returned error can't find the container with id 99e38e60a9b6043d3b782294240b61b4bfbd8f8811a9070f0a4865a7a9fe2cfa Aug 01 00:39:03.597237 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:03.597200 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:39:03.726474 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:03.726441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" event={"ID":"bbe12152-f594-4f89-85a0-26f490f11c97","Type":"ContainerStarted","Data":"99e38e60a9b6043d3b782294240b61b4bfbd8f8811a9070f0a4865a7a9fe2cfa"} Aug 01 00:39:04.732339 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:04.732303 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" event={"ID":"bbe12152-f594-4f89-85a0-26f490f11c97","Type":"ContainerStarted","Data":"cbd4c3f4cf225fcf35c5d9ef770949238f79b9baab9b5d08b5be326dd9d58e11"} Aug 01 00:39:04.751175 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:04.751112 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" podStartSLOduration=1.7762937989999998 podStartE2EDuration="2.751094747s" totalImagesPullingTime="974.800948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:04.749775245 +0000 UTC m=+594.106566902" watchObservedRunningTime="2026-08-01 00:39:04.751094747 +0000 UTC m=+594.107886403" Aug 01 00:39:08.575512 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:08.575474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:39:10.784754 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:10.784714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:39:11.080916 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:11.080836 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:39:11.080916 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:11.080904 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:39:12.760075 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:12.760037 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:12.760554 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:12.760104 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:12.773241 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:12.773207 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:12.784431 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:12.784409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft" Aug 01 00:39:16.140561 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.140530 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf"] Aug 01 00:39:16.143781 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.143765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.146301 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.146277 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 01 00:39:16.155971 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.155942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf"] Aug 01 00:39:16.167073 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.167041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:39:16.244853 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.244810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8nqs\" (UniqueName: \"kubernetes.io/projected/e9666785-eedd-4184-8cc0-c8f663665f8d-kube-api-access-g8nqs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.245058 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.244858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.245058 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.244921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9666785-eedd-4184-8cc0-c8f663665f8d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.245058 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.244963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.245058 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.244984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.245058 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.245018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346319 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nqs\" (UniqueName: \"kubernetes.io/projected/e9666785-eedd-4184-8cc0-c8f663665f8d-kube-api-access-g8nqs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346319 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9666785-eedd-4184-8cc0-c8f663665f8d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346587 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346801 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346777 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346850 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.346889 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.346852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.348601 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.348581 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9666785-eedd-4184-8cc0-c8f663665f8d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.348947 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.348930 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9666785-eedd-4184-8cc0-c8f663665f8d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.354510 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.354490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8nqs\" (UniqueName: \"kubernetes.io/projected/e9666785-eedd-4184-8cc0-c8f663665f8d-kube-api-access-g8nqs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q85gf\" (UID: \"e9666785-eedd-4184-8cc0-c8f663665f8d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.453000 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.452962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:16.586691 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.586649 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf"] Aug 01 00:39:16.590240 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:39:16.590213 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9666785_eedd_4184_8cc0_c8f663665f8d.slice/crio-a37c699aec7a0ba91605635b9877896983f13af99cad20be767160613c20cb41 WatchSource:0}: Error finding container a37c699aec7a0ba91605635b9877896983f13af99cad20be767160613c20cb41: Status 404 returned error can't find the container with id a37c699aec7a0ba91605635b9877896983f13af99cad20be767160613c20cb41 Aug 01 00:39:16.791146 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:16.791113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" event={"ID":"e9666785-eedd-4184-8cc0-c8f663665f8d","Type":"ContainerStarted","Data":"a37c699aec7a0ba91605635b9877896983f13af99cad20be767160613c20cb41"} Aug 01 00:39:17.797786 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:17.797752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" event={"ID":"e9666785-eedd-4184-8cc0-c8f663665f8d","Type":"ContainerStarted","Data":"95497a5effea7cae6cf9ff18c8a8c669af55209cc1bf5b97d88e2b1f394982e0"} Aug 01 00:39:17.825121 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:17.825064 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" podStartSLOduration=1.6726690309999999 podStartE2EDuration="1.825046147s" totalImagesPullingTime="152.377116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:17.823573992 +0000 UTC m=+607.180365649" watchObservedRunningTime="2026-08-01 00:39:17.825046147 +0000 UTC m=+607.181837803" Aug 01 00:39:18.478243 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:18.478209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:39:26.453941 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:26.453902 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:26.454377 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:26.453992 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:26.466731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:26.466703 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:39:26.848686 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:39:26.848611 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q85gf" Aug 01 00:40:05.764498 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.764461 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77fd747845-ddj76"] Aug 01 00:40:05.769753 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.768638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:05.773457 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.773434 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vpbxr\"" Aug 01 00:40:05.773690 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.773659 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 01 00:40:05.776568 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.776541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77fd747845-ddj76"] Aug 01 00:40:05.836245 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.836202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4134e1dd-a10f-4323-a2e9-0fab9f6779d9-tls-cert\") pod \"authorino-77fd747845-ddj76\" (UID: \"4134e1dd-a10f-4323-a2e9-0fab9f6779d9\") " pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:05.836245 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.836245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6kf\" (UniqueName: \"kubernetes.io/projected/4134e1dd-a10f-4323-a2e9-0fab9f6779d9-kube-api-access-2k6kf\") pod \"authorino-77fd747845-ddj76\" (UID: \"4134e1dd-a10f-4323-a2e9-0fab9f6779d9\") " pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:05.937548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.937508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4134e1dd-a10f-4323-a2e9-0fab9f6779d9-tls-cert\") pod \"authorino-77fd747845-ddj76\" (UID: \"4134e1dd-a10f-4323-a2e9-0fab9f6779d9\") " pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:05.937548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.937549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6kf\" (UniqueName: \"kubernetes.io/projected/4134e1dd-a10f-4323-a2e9-0fab9f6779d9-kube-api-access-2k6kf\") pod \"authorino-77fd747845-ddj76\" (UID: \"4134e1dd-a10f-4323-a2e9-0fab9f6779d9\") " pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:05.939927 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.939906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4134e1dd-a10f-4323-a2e9-0fab9f6779d9-tls-cert\") pod \"authorino-77fd747845-ddj76\" (UID: \"4134e1dd-a10f-4323-a2e9-0fab9f6779d9\") " pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:05.944644 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:05.944622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6kf\" (UniqueName: \"kubernetes.io/projected/4134e1dd-a10f-4323-a2e9-0fab9f6779d9-kube-api-access-2k6kf\") pod \"authorino-77fd747845-ddj76\" (UID: \"4134e1dd-a10f-4323-a2e9-0fab9f6779d9\") " pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:06.080958 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:06.080854 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fd747845-ddj76" Aug 01 00:40:06.211279 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:06.211256 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77fd747845-ddj76"] Aug 01 00:40:06.213321 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:40:06.213287 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4134e1dd_a10f_4323_a2e9_0fab9f6779d9.slice/crio-fd1607852f145e5aedd14e42ceaa0cc317e31bc696b26cf68189567d99402d96 WatchSource:0}: Error finding container fd1607852f145e5aedd14e42ceaa0cc317e31bc696b26cf68189567d99402d96: Status 404 returned error can't find the container with id fd1607852f145e5aedd14e42ceaa0cc317e31bc696b26cf68189567d99402d96 Aug 01 00:40:06.214740 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:06.214716 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 00:40:07.025909 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:07.025873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fd747845-ddj76" event={"ID":"4134e1dd-a10f-4323-a2e9-0fab9f6779d9","Type":"ContainerStarted","Data":"fd1607852f145e5aedd14e42ceaa0cc317e31bc696b26cf68189567d99402d96"} Aug 01 00:40:08.035207 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:08.035169 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fd747845-ddj76" event={"ID":"4134e1dd-a10f-4323-a2e9-0fab9f6779d9","Type":"ContainerStarted","Data":"aef48d1c05f2cc1a02ae51afd7b2a7c00019ebba11d99d48c64907ac9dfe9a5a"} Aug 01 00:40:08.057833 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:08.057768 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77fd747845-ddj76" podStartSLOduration=2.31449209 podStartE2EDuration="3.057747856s" totalImagesPullingTime="743.255766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:40:08.052843153 +0000 UTC m=+657.409634816" watchObservedRunningTime="2026-08-01 00:40:08.057747856 +0000 UTC m=+657.414539516" Aug 01 00:40:08.565511 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:08.565477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:40:08.667150 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:08.667107 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:40:09.646113 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:09.646078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:40:09.745637 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:09.745593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:40:22.990518 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:22.990480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:40:37.076219 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:37.076173 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:40:45.870666 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:45.870629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:40:56.876151 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:40:56.876113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:42:13.805710 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:42:13.805666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:42:15.006177 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:42:15.006123 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:42:38.063240 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:42:38.063150 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:42:39.368849 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:42:39.368809 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:42:48.570071 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:42:48.570030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:43:19.365800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:19.365761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:19.502875 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:19.502846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:19.727950 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:19.727911 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:19.826867 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:19.826831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:19.926907 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:19.926876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:25.535926 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:25.535887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:25.602476 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:25.602443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:25.668473 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:25.668439 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:25.771686 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:25.771653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:25.939102 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:25.939066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:31.727077 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:31.727036 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:31.804069 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:31.804034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:43:47.368506 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:47.368467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:43:54.388029 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:54.387995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:43:57.528751 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:43:57.528718 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:44:09.872477 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:09.872440 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:44:11.110488 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:11.110460 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:44:11.110980 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:11.110964 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:44:15.466951 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:15.466917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:44:25.467501 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:25.467457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:44:31.568887 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:31.568847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:44:45.778466 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:45.778425 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:44:50.868756 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:50.868718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:44:55.873310 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:44:55.873277 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:45:02.574194 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:45:02.574141 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:45:08.565275 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:45:08.565233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:45:20.667427 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:45:20.667392 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:45:31.372545 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:45:31.372512 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:45:53.769876 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:45:53.769834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:04.076471 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:04.076435 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:09.974354 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:09.974316 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:17.075743 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:17.075702 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:24.177720 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:24.177679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:29.674133 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:29.674094 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:31.957581 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:31.957544 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:46:34.978036 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:34.978000 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:46:35.700717 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:35.700671 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:42.861457 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:42.861416 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:46:54.380964 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:46:54.380924 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:47:00.964108 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:00.964069 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:47:12.267402 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:12.267364 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:47:16.482769 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:16.482736 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:47:22.372815 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:22.372776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:47:32.364678 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:32.364643 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:47:37.372542 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:37.372504 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:47:43.710361 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:43.710327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:47:48.074704 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:47:48.074666 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:49:00.640407 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:49:00.640371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:49:11.140536 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:49:11.140505 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:49:11.141253 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:49:11.141234 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:50:04.070843 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:50:04.070807 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:50:09.170945 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:50:09.170908 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:50:35.463221 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:50:35.463190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:50:55.275731 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:50:55.275692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:05.576506 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:05.576469 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:10.760666 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:10.760619 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:16.970045 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:16.970003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:25.771961 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:25.771922 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:30.572654 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:30.572612 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:40.881855 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:40.881772 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:46.475558 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:46.475526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:51:56.169520 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:51:56.169480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:52:02.567266 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:52:02.567222 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:52:11.774021 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:52:11.773982 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:52:36.475117 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:52:36.475076 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:52:41.965105 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:52:41.965069 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:53:07.852285 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.852246 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz"] Aug 01 00:53:07.855551 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.855533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:07.858064 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.858044 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-886403a4-kserve-self-signed-certs\"" Aug 01 00:53:07.873777 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.873748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz"] Aug 01 00:53:07.911665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.911629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-home\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:07.911665 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.911667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-model-cache\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:07.911876 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.911714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8940cdc-3710-47d4-902a-35ae21c4a45a-tls-certs\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:07.911876 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.911761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-dshm\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:07.911876 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.911798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5t4\" (UniqueName: \"kubernetes.io/projected/e8940cdc-3710-47d4-902a-35ae21c4a45a-kube-api-access-wt5t4\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:07.911876 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.911863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-tmp-dir\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:07.918373 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:07.918341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:53:08.012813 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.012771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8940cdc-3710-47d4-902a-35ae21c4a45a-tls-certs\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.012813 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.012811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-dshm\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.013048 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.012830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5t4\" (UniqueName: \"kubernetes.io/projected/e8940cdc-3710-47d4-902a-35ae21c4a45a-kube-api-access-wt5t4\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.013048 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.012864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-tmp-dir\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.013048 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.012899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-home\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.013048 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.012952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-model-cache\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.013381 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.013359 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-home\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.013455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.013364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-tmp-dir\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.013455 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.013399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-model-cache\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.015215 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.015194 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-dshm\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.015370 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.015353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8940cdc-3710-47d4-902a-35ae21c4a45a-tls-certs\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.022176 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.022117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5t4\" (UniqueName: \"kubernetes.io/projected/e8940cdc-3710-47d4-902a-35ae21c4a45a-kube-api-access-wt5t4\") pod \"e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.165602 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.165566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:08.300124 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.300099 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz"] Aug 01 00:53:08.302037 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:53:08.302010 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8940cdc_3710_47d4_902a_35ae21c4a45a.slice/crio-4a53e5219a4f1007172c92b065a7f16c31838bf7b47dbe93bb632b207c24756f WatchSource:0}: Error finding container 4a53e5219a4f1007172c92b065a7f16c31838bf7b47dbe93bb632b207c24756f: Status 404 returned error can't find the container with id 4a53e5219a4f1007172c92b065a7f16c31838bf7b47dbe93bb632b207c24756f Aug 01 00:53:08.668840 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.668799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" event={"ID":"e8940cdc-3710-47d4-902a-35ae21c4a45a","Type":"ContainerStarted","Data":"dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349"} Aug 01 00:53:08.669047 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.668858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" event={"ID":"e8940cdc-3710-47d4-902a-35ae21c4a45a","Type":"ContainerStarted","Data":"4a53e5219a4f1007172c92b065a7f16c31838bf7b47dbe93bb632b207c24756f"} Aug 01 00:53:08.685473 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:08.685420 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" podStartSLOduration=1.685402555 podStartE2EDuration="1.685402555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:53:08.684346325 +0000 UTC m=+1438.041137972" watchObservedRunningTime="2026-08-01 00:53:08.685402555 +0000 UTC m=+1438.042194189" Aug 01 00:53:10.770726 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:10.770696 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:53:18.165812 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:18.165768 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:18.166268 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:18.165832 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:18.178345 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:18.178319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:18.726668 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:18.726640 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:53:23.140233 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.140196 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd"] Aug 01 00:53:23.143727 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.143709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.146001 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.145984 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-886403a4-kserve-self-signed-certs\"" Aug 01 00:53:23.150236 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.150214 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd"] Aug 01 00:53:23.234406 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.234375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:53:23.258405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.258365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tmp-dir\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.258405 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.258402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwk68\" (UniqueName: \"kubernetes.io/projected/2b0a0ccc-72da-4689-a5c1-b173686c1db3-kube-api-access-wwk68\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.258634 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.258431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-model-cache\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.258634 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.258520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-dshm\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.258634 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.258557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-home\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.258634 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.258620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tls-certs\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360014 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.359974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tmp-dir\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360178 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwk68\" (UniqueName: \"kubernetes.io/projected/2b0a0ccc-72da-4689-a5c1-b173686c1db3-kube-api-access-wwk68\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360178 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-model-cache\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360178 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-dshm\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360178 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-home\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360342 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tls-certs\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360492 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tmp-dir\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360543 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-model-cache\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.360543 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.360532 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-home\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.362503 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.362484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-dshm\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.362769 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.362753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tls-certs\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.367512 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.367488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwk68\" (UniqueName: \"kubernetes.io/projected/2b0a0ccc-72da-4689-a5c1-b173686c1db3-kube-api-access-wwk68\") pod \"e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.453523 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.453481 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:23.587752 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.587726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd"] Aug 01 00:53:23.590292 ip-10-0-140-111 kubenswrapper[2575]: W0801 00:53:23.590265 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b0a0ccc_72da_4689_a5c1_b173686c1db3.slice/crio-ecc21e5df844309dc5845f6574de4b0dd223a90b9cbd87b431d2b903672311af WatchSource:0}: Error finding container ecc21e5df844309dc5845f6574de4b0dd223a90b9cbd87b431d2b903672311af: Status 404 returned error can't find the container with id ecc21e5df844309dc5845f6574de4b0dd223a90b9cbd87b431d2b903672311af Aug 01 00:53:23.740788 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.740696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" event={"ID":"2b0a0ccc-72da-4689-a5c1-b173686c1db3","Type":"ContainerStarted","Data":"454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4"} Aug 01 00:53:23.740788 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.740738 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" event={"ID":"2b0a0ccc-72da-4689-a5c1-b173686c1db3","Type":"ContainerStarted","Data":"ecc21e5df844309dc5845f6574de4b0dd223a90b9cbd87b431d2b903672311af"} Aug 01 00:53:23.758813 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:23.758762 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" podStartSLOduration=0.758745234 podStartE2EDuration="758.745234ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:53:23.756025513 +0000 UTC m=+1453.112817170" watchObservedRunningTime="2026-08-01 00:53:23.758745234 +0000 UTC m=+1453.115536889" Aug 01 00:53:25.681111 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:25.681067 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:53:33.454298 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:33.454252 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:33.454298 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:33.454305 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:33.466738 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:33.466706 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:53:33.798120 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:53:33.798033 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:54:03.200112 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:03.200079 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz"] Aug 01 00:54:04.012776 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:04.012744 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:54:04.973036 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:04.973002 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:54:05.028201 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:05.028150 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd"] Aug 01 00:54:05.046414 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:05.046385 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:54:11.175089 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:11.175058 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:54:11.176450 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:11.176426 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:54:16.261966 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:16.261935 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:54:18.219211 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.219137 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" podUID="e8940cdc-3710-47d4-902a-35ae21c4a45a" containerName="main" containerID="cri-o://dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349" gracePeriod=45 Aug 01 00:54:18.462975 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.462947 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:54:18.634218 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634103 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8940cdc-3710-47d4-902a-35ae21c4a45a-tls-certs\") pod \"e8940cdc-3710-47d4-902a-35ae21c4a45a\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " Aug 01 00:54:18.634218 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634199 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-home\") pod \"e8940cdc-3710-47d4-902a-35ae21c4a45a\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " Aug 01 00:54:18.634419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634305 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-model-cache\") pod \"e8940cdc-3710-47d4-902a-35ae21c4a45a\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " Aug 01 00:54:18.634419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634341 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt5t4\" (UniqueName: \"kubernetes.io/projected/e8940cdc-3710-47d4-902a-35ae21c4a45a-kube-api-access-wt5t4\") pod \"e8940cdc-3710-47d4-902a-35ae21c4a45a\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " Aug 01 00:54:18.634419 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634358 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-tmp-dir\") pod \"e8940cdc-3710-47d4-902a-35ae21c4a45a\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " Aug 01 00:54:18.634561 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634427 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-home" (OuterVolumeSpecName: "home") pod "e8940cdc-3710-47d4-902a-35ae21c4a45a" (UID: "e8940cdc-3710-47d4-902a-35ae21c4a45a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:18.634561 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634436 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-dshm\") pod \"e8940cdc-3710-47d4-902a-35ae21c4a45a\" (UID: \"e8940cdc-3710-47d4-902a-35ae21c4a45a\") " Aug 01 00:54:18.634848 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.634796 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8940cdc-3710-47d4-902a-35ae21c4a45a" (UID: "e8940cdc-3710-47d4-902a-35ae21c4a45a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:18.635221 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.635172 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8940cdc-3710-47d4-902a-35ae21c4a45a" (UID: "e8940cdc-3710-47d4-902a-35ae21c4a45a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:18.635753 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.635727 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:18.635850 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.635760 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:18.635850 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.635778 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:18.637250 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.637215 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-dshm" (OuterVolumeSpecName: "dshm") pod "e8940cdc-3710-47d4-902a-35ae21c4a45a" (UID: "e8940cdc-3710-47d4-902a-35ae21c4a45a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:18.637344 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.637288 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8940cdc-3710-47d4-902a-35ae21c4a45a-kube-api-access-wt5t4" (OuterVolumeSpecName: "kube-api-access-wt5t4") pod "e8940cdc-3710-47d4-902a-35ae21c4a45a" (UID: "e8940cdc-3710-47d4-902a-35ae21c4a45a"). InnerVolumeSpecName "kube-api-access-wt5t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:54:18.639314 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.638275 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8940cdc-3710-47d4-902a-35ae21c4a45a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8940cdc-3710-47d4-902a-35ae21c4a45a" (UID: "e8940cdc-3710-47d4-902a-35ae21c4a45a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:54:18.736505 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.736466 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt5t4\" (UniqueName: \"kubernetes.io/projected/e8940cdc-3710-47d4-902a-35ae21c4a45a-kube-api-access-wt5t4\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:18.736505 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.736499 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8940cdc-3710-47d4-902a-35ae21c4a45a-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:18.736505 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:18.736511 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8940cdc-3710-47d4-902a-35ae21c4a45a-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:19.001196 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.001147 2575 generic.go:356] "Generic (PLEG): container finished" podID="e8940cdc-3710-47d4-902a-35ae21c4a45a" containerID="dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349" exitCode=0 Aug 01 00:54:19.001375 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.001232 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" Aug 01 00:54:19.001375 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.001235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" event={"ID":"e8940cdc-3710-47d4-902a-35ae21c4a45a","Type":"ContainerDied","Data":"dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349"} Aug 01 00:54:19.001375 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.001347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz" event={"ID":"e8940cdc-3710-47d4-902a-35ae21c4a45a","Type":"ContainerDied","Data":"4a53e5219a4f1007172c92b065a7f16c31838bf7b47dbe93bb632b207c24756f"} Aug 01 00:54:19.001375 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.001368 2575 scope.go:117] "RemoveContainer" containerID="dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349" Aug 01 00:54:19.010725 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.010708 2575 scope.go:117] "RemoveContainer" containerID="dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349" Aug 01 00:54:19.010987 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:54:19.010968 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349\": container with ID starting with dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349 not found: ID does not exist" containerID="dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349" Aug 01 00:54:19.011047 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.010995 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349"} err="failed to get container status \"dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349\": rpc error: code = NotFound desc = could not find container \"dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349\": container with ID starting with dfa37bc1f4c9e90749c9966aa8a2a5ed3d882bf6e7286598ea614112d253c349 not found: ID does not exist" Aug 01 00:54:19.022960 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.022928 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz"] Aug 01 00:54:19.028407 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.028375 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-886403a4-kserve-88b56d58c-vszjz"] Aug 01 00:54:19.163108 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:19.163074 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8940cdc-3710-47d4-902a-35ae21c4a45a" path="/var/lib/kubelet/pods/e8940cdc-3710-47d4-902a-35ae21c4a45a/volumes" Aug 01 00:54:20.047192 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.047132 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" podUID="2b0a0ccc-72da-4689-a5c1-b173686c1db3" containerName="main" containerID="cri-o://454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4" gracePeriod=45 Aug 01 00:54:20.297125 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.297099 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:54:20.451283 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451244 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwk68\" (UniqueName: \"kubernetes.io/projected/2b0a0ccc-72da-4689-a5c1-b173686c1db3-kube-api-access-wwk68\") pod \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " Aug 01 00:54:20.451283 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451291 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tmp-dir\") pod \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " Aug 01 00:54:20.451548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451343 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-home\") pod \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " Aug 01 00:54:20.451548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451361 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tls-certs\") pod \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " Aug 01 00:54:20.451548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451392 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-model-cache\") pod \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " Aug 01 00:54:20.451548 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451411 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-dshm\") pod \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\" (UID: \"2b0a0ccc-72da-4689-a5c1-b173686c1db3\") " Aug 01 00:54:20.451800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451638 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b0a0ccc-72da-4689-a5c1-b173686c1db3" (UID: "2b0a0ccc-72da-4689-a5c1-b173686c1db3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:20.451800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451659 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-home" (OuterVolumeSpecName: "home") pod "2b0a0ccc-72da-4689-a5c1-b173686c1db3" (UID: "2b0a0ccc-72da-4689-a5c1-b173686c1db3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:20.451800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.451678 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b0a0ccc-72da-4689-a5c1-b173686c1db3" (UID: "2b0a0ccc-72da-4689-a5c1-b173686c1db3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:20.453519 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.453493 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-dshm" (OuterVolumeSpecName: "dshm") pod "2b0a0ccc-72da-4689-a5c1-b173686c1db3" (UID: "2b0a0ccc-72da-4689-a5c1-b173686c1db3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 00:54:20.453622 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.453540 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b0a0ccc-72da-4689-a5c1-b173686c1db3" (UID: "2b0a0ccc-72da-4689-a5c1-b173686c1db3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 00:54:20.453622 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.453577 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0a0ccc-72da-4689-a5c1-b173686c1db3-kube-api-access-wwk68" (OuterVolumeSpecName: "kube-api-access-wwk68") pod "2b0a0ccc-72da-4689-a5c1-b173686c1db3" (UID: "2b0a0ccc-72da-4689-a5c1-b173686c1db3"). InnerVolumeSpecName "kube-api-access-wwk68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 00:54:20.552564 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.552523 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:20.552564 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.552556 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:20.552564 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.552565 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:20.552564 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.552574 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwk68\" (UniqueName: \"kubernetes.io/projected/2b0a0ccc-72da-4689-a5c1-b173686c1db3-kube-api-access-wwk68\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:20.552826 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.552583 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:20.552826 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:20.552594 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b0a0ccc-72da-4689-a5c1-b173686c1db3-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 00:54:21.012603 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.012570 2575 generic.go:356] "Generic (PLEG): container finished" podID="2b0a0ccc-72da-4689-a5c1-b173686c1db3" containerID="454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4" exitCode=0 Aug 01 00:54:21.012772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.012616 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" Aug 01 00:54:21.012772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.012659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" event={"ID":"2b0a0ccc-72da-4689-a5c1-b173686c1db3","Type":"ContainerDied","Data":"454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4"} Aug 01 00:54:21.012772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.012703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd" event={"ID":"2b0a0ccc-72da-4689-a5c1-b173686c1db3","Type":"ContainerDied","Data":"ecc21e5df844309dc5845f6574de4b0dd223a90b9cbd87b431d2b903672311af"} Aug 01 00:54:21.012772 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.012725 2575 scope.go:117] "RemoveContainer" containerID="454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4" Aug 01 00:54:21.021686 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.021666 2575 scope.go:117] "RemoveContainer" containerID="454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4" Aug 01 00:54:21.021960 ip-10-0-140-111 kubenswrapper[2575]: E0801 00:54:21.021941 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4\": container with ID starting with 454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4 not found: ID does not exist" containerID="454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4" Aug 01 00:54:21.022008 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.021970 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4"} err="failed to get container status \"454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4\": rpc error: code = NotFound desc = could not find container \"454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4\": container with ID starting with 454c5d7f3d35bbb2d4c53eb9de80757eb1a2f8d6ff60dab0aba21ae626cbfbb4 not found: ID does not exist" Aug 01 00:54:21.032735 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.032699 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd"] Aug 01 00:54:21.037391 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.037367 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-886403a4-kserve-5f775dbbf9-zx7fd"] Aug 01 00:54:21.162920 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.162877 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b0a0ccc-72da-4689-a5c1-b173686c1db3" path="/var/lib/kubelet/pods/2b0a0ccc-72da-4689-a5c1-b173686c1db3/volumes" Aug 01 00:54:21.167710 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:21.167684 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:54:30.170763 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:30.170722 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:54:47.669436 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:47.669362 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:54:54.881865 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:54:54.881829 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:55:01.408558 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:55:01.408515 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:55:09.675252 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:55:09.675207 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:55:19.768082 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:55:19.768038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:55:27.273297 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:55:27.273258 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:55:44.565579 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:55:44.565545 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:55:50.069388 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:55:50.069353 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:00.872856 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:00.872815 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:08.463822 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:08.463784 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:09.190479 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:09.190445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:56:14.872836 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:14.872800 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:19.069229 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:19.069189 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:26.172579 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:26.172539 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:35.468796 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:35.468762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:43.075732 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:43.075680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:52.478779 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:52.478743 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:56:59.570537 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:56:59.570492 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:57:09.369817 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:09.369784 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:57:24.566972 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:24.566934 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:57:30.361176 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:30.361110 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:57:33.946209 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:33.946101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:57:36.577246 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:36.577214 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:57:48.271337 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:48.271299 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:57:56.570390 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:56.570355 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:57:59.614740 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:57:59.614705 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:02.167326 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:02.167285 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:58:02.528418 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:02.528308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:02.629540 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:02.629503 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:02.937711 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:02.937672 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:08.580406 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:08.580367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:08.686379 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:08.686347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:08.847662 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:08.847572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:09.112297 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:09.112222 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:10.390800 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:10.390767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:11.833626 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:11.833578 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:11.925483 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:11.925443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:13.066889 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:13.066857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:13.522492 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:13.522457 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:13.617408 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:13.617376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:16.063353 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:16.063320 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:58:43.539135 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:43.539101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:43.617712 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:43.617679 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:43.677838 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:43.677796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:43.739396 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:43.739363 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:43.861059 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:43.860953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:44.065363 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:44.065324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:44.299994 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:44.299962 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:44.398834 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:44.398795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:44.556637 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:44.556545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:44.822195 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:44.822081 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:45.200951 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:45.200908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:46.864484 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:46.864449 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:58:50.440601 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:50.440550 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:50.545976 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:50.545943 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:50.793253 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:50.793141 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:51.172338 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:51.172298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:51.272850 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:51.272816 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:58:54.762502 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:58:54.762467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:59:11.204659 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:11.204628 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:59:11.206540 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:11.206519 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 00:59:21.079149 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:21.079107 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:21.146039 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:21.146002 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:21.240140 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:21.240099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:21.301767 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:21.301714 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:21.386903 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:21.386824 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:21.595813 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:21.595782 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:21.862626 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:21.862591 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:22.143068 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:22.142981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:22.580034 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:22.580002 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:24.068920 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:24.068885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:59:24.542912 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:24.542875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:24.643250 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:24.643216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:25.810577 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:25.810543 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:26.024819 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:26.024783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:26.273807 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:26.273771 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:26.374623 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:26.374586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:33.789442 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:33.789404 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 00:59:56.164794 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:56.164754 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:56.265266 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:56.265229 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:56.338287 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:56.338249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:56.400459 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:56.400427 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:57.451065 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:57.451034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:57.662353 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:57.662305 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:57.861949 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:57.861857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:58.162487 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:58.162453 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:58.374172 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:58.374136 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:58.459320 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:58.459226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:58.517570 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:58.517533 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 00:59:59.164617 ip-10-0-140-111 kubenswrapper[2575]: I0801 00:59:59.164586 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:00:00.759511 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:00.759477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:00.857578 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:00.857542 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:01.887706 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:01.887670 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:02.056009 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:02.055976 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:02.489073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:02.489041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:02.688572 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:02.688536 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:05.385603 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:05.385555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:00:32.349357 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:32.349275 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:32.416385 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:32.416350 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:32.491873 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:32.491825 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:32.550514 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:32.550478 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:32.671000 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:32.670960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:32.896793 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:32.896762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:33.322406 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:33.322371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:33.422736 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:33.422674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:33.530994 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:33.530951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:35.274985 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:35.274946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:00:45.143203 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:45.143144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:48.165220 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:48.165188 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:00:51.658517 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:51.658481 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:51.767376 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:51.767341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:51.913504 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:51.913423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:52.013959 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:52.013928 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:00:52.395436 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:00:52.395328 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:18.310800 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:18.310691 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:24.313056 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:24.313019 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:24.418095 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:24.418061 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:24.590541 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:24.590432 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:40.345632 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:40.345590 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:46.209344 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:46.209308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:46.319095 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:46.319048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:46.410761 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:46.410702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:01:56.875367 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:01:56.875327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:07.908813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:07.908730 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:13.675943 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:13.675889 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:13.779901 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:13.779864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:13.876143 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:13.876079 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:13.980920 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:13.980829 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:14.614440 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:14.614409 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:14.713819 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:14.713763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:14.813797 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:14.813760 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:15.566652 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:15.566615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:02:16.204732 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:16.204695 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:22.997629 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:22.997594 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:23.097235 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:23.097198 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:23.646557 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:23.646521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:24.280940 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:24.280904 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:02:29.599076 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:29.599040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:29.703964 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:29.703932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:29.829032 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:29.828994 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:30.233070 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:30.233035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:30.418470 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:30.418438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:33.054005 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:33.053975 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:33.219277 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:33.219242 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:34.235219 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:34.235182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:34.413602 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:34.413569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:34.818833 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:34.818796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:34.915232 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:34.915200 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:37.172856 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:37.172820 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:02:58.013991 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:58.013951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:58.726915 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:58.726877 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:58.846122 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:58.846085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:02:59.177776 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:02:59.177704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:05.093606 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:05.093565 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:05.196253 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:05.196218 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:12.785546 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:12.785516 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:16.035793 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:16.035755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:16.122377 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:16.122336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:16.201814 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:16.201780 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:16.295866 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:16.295787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:16.644319 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:16.644229 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:18.273209 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:18.273172 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:03:23.159022 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:23.158992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:23.261764 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:23.261725 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:23.444670 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:23.444572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:23.680758 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:23.680723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:23.788505 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:23.788423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:23.879803 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:23.879770 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:24.016530 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:24.016496 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:29.759139 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:29.759101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:29.865688 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:29.865645 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:29.959467 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:29.959428 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:30.097003 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:30.096912 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:35.966808 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:35.966728 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:36.094376 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:36.094345 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:03:57.382177 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:03:57.379060 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:04:11.238768 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:04:11.238731 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:04:11.241123 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:04:11.241101 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:04:38.853683 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:04:38.853636 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:04:42.270688 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:04:42.270638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:15.990578 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:15.990547 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:38.448087 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:38.448055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:38.757210 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:38.757102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:38.958681 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:38.958648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:40.571397 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:40.571311 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:05:41.976037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:41.975999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:42.075792 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:42.075753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:43.087484 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:43.087435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:43.337177 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:43.337122 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:43.619535 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:43.619480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:43.702786 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:43.702751 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:05:51.280191 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:05:51.280133 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:06:06.897397 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:06.897364 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:06:18.338697 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.338664 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x"] Aug 01 01:06:18.339215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.338982 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0a0ccc-72da-4689-a5c1-b173686c1db3" containerName="main" Aug 01 01:06:18.339215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.338994 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0a0ccc-72da-4689-a5c1-b173686c1db3" containerName="main" Aug 01 01:06:18.339215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.339003 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8940cdc-3710-47d4-902a-35ae21c4a45a" containerName="main" Aug 01 01:06:18.339215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.339008 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8940cdc-3710-47d4-902a-35ae21c4a45a" containerName="main" Aug 01 01:06:18.339215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.339063 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0a0ccc-72da-4689-a5c1-b173686c1db3" containerName="main" Aug 01 01:06:18.339215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.339073 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8940cdc-3710-47d4-902a-35ae21c4a45a" containerName="main" Aug 01 01:06:18.341907 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.341888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.344193 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.344171 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e23e15a8\"/\"openshift-service-ca.crt\"" Aug 01 01:06:18.344307 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.344171 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e23e15a8\"/\"default-dockercfg-d58s8\"" Aug 01 01:06:18.345066 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.345049 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e23e15a8\"/\"kube-root-ca.crt\"" Aug 01 01:06:18.345109 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.345089 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e23e15a8\"/\"e2e-lifecycle-model-e23e15a8-kserve-self-signed-certs\"" Aug 01 01:06:18.351672 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.351647 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x"] Aug 01 01:06:18.364679 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.364652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:06:18.391039 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.390984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-model-cache\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.391039 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.391040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tmp-dir\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.391276 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.391070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-home\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.391276 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.391109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqhk8\" (UniqueName: \"kubernetes.io/projected/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-kube-api-access-jqhk8\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.391276 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.391139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-dshm\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.391276 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.391228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tls-certs\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.491655 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.491621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqhk8\" (UniqueName: \"kubernetes.io/projected/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-kube-api-access-jqhk8\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.491655 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.491663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-dshm\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.491937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.491694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tls-certs\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.491937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.491721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-model-cache\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.491937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.491740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tmp-dir\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.491937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.491763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-home\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.492199 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.492145 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-home\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.492273 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.492217 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tmp-dir\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.492273 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.492211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-model-cache\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.494028 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.494011 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-dshm\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.494205 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.494187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tls-certs\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.501223 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.501201 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqhk8\" (UniqueName: \"kubernetes.io/projected/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-kube-api-access-jqhk8\") pod \"e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.651457 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.651368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:18.787734 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:18.787699 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x"] Aug 01 01:06:18.789898 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:06:18.789869 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda568be4e_e6bb_4fd9_9328_be365f5fa0d6.slice/crio-0768a8a57c7da4b46b02f4a15f40a9714d60844416862a45c71da2dec884ec0c WatchSource:0}: Error finding container 0768a8a57c7da4b46b02f4a15f40a9714d60844416862a45c71da2dec884ec0c: Status 404 returned error can't find the container with id 0768a8a57c7da4b46b02f4a15f40a9714d60844416862a45c71da2dec884ec0c Aug 01 01:06:19.328751 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:19.328709 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" event={"ID":"a568be4e-e6bb-4fd9-9328-be365f5fa0d6","Type":"ContainerStarted","Data":"2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c"} Aug 01 01:06:19.328751 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:19.328750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" event={"ID":"a568be4e-e6bb-4fd9-9328-be365f5fa0d6","Type":"ContainerStarted","Data":"0768a8a57c7da4b46b02f4a15f40a9714d60844416862a45c71da2dec884ec0c"} Aug 01 01:06:19.345392 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:19.345342 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" podStartSLOduration=1.345326203 podStartE2EDuration="1.345326203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:06:19.343780283 +0000 UTC m=+2228.700571940" watchObservedRunningTime="2026-08-01 01:06:19.345326203 +0000 UTC m=+2228.702117860" Aug 01 01:06:20.760657 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:20.760619 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:06:28.652227 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:28.652187 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:28.652711 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:28.652249 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:28.664998 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:28.664965 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:29.390260 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:29.390228 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:06:37.568353 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:37.568265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:06:59.274094 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:06:59.274060 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:07:01.828060 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:01.828026 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:01.900367 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:01.900332 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:02.000294 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:02.000259 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:02.066217 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:02.066180 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:04.967713 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:04.967676 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:07:05.067938 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:05.067902 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:10.048606 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:10.048565 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x"] Aug 01 01:07:10.144688 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:10.144655 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:25.067471 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.067423 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" podUID="a568be4e-e6bb-4fd9-9328-be365f5fa0d6" containerName="main" containerID="cri-o://2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c" gracePeriod=45 Aug 01 01:07:25.313801 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.313775 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:07:25.327758 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.327675 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tmp-dir\") pod \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " Aug 01 01:07:25.327758 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.327737 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqhk8\" (UniqueName: \"kubernetes.io/projected/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-kube-api-access-jqhk8\") pod \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " Aug 01 01:07:25.327758 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.327760 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tls-certs\") pod \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " Aug 01 01:07:25.328003 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.327803 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-model-cache\") pod \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " Aug 01 01:07:25.328003 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.327875 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-home\") pod \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " Aug 01 01:07:25.328003 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.327899 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-dshm\") pod \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\" (UID: \"a568be4e-e6bb-4fd9-9328-be365f5fa0d6\") " Aug 01 01:07:25.328003 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.327954 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a568be4e-e6bb-4fd9-9328-be365f5fa0d6" (UID: "a568be4e-e6bb-4fd9-9328-be365f5fa0d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:25.328226 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.328063 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a568be4e-e6bb-4fd9-9328-be365f5fa0d6" (UID: "a568be4e-e6bb-4fd9-9328-be365f5fa0d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:25.328226 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.328116 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:07:25.328226 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.328135 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:07:25.328226 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.328177 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-home" (OuterVolumeSpecName: "home") pod "a568be4e-e6bb-4fd9-9328-be365f5fa0d6" (UID: "a568be4e-e6bb-4fd9-9328-be365f5fa0d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:25.330131 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.330065 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-dshm" (OuterVolumeSpecName: "dshm") pod "a568be4e-e6bb-4fd9-9328-be365f5fa0d6" (UID: "a568be4e-e6bb-4fd9-9328-be365f5fa0d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:07:25.330284 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.330193 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-kube-api-access-jqhk8" (OuterVolumeSpecName: "kube-api-access-jqhk8") pod "a568be4e-e6bb-4fd9-9328-be365f5fa0d6" (UID: "a568be4e-e6bb-4fd9-9328-be365f5fa0d6"). InnerVolumeSpecName "kube-api-access-jqhk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:07:25.330284 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.330231 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a568be4e-e6bb-4fd9-9328-be365f5fa0d6" (UID: "a568be4e-e6bb-4fd9-9328-be365f5fa0d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:07:25.428914 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.428871 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:07:25.428914 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.428905 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:07:25.428914 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.428917 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqhk8\" (UniqueName: \"kubernetes.io/projected/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-kube-api-access-jqhk8\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:07:25.429204 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.428928 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a568be4e-e6bb-4fd9-9328-be365f5fa0d6-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:07:25.643906 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.643810 2575 generic.go:356] "Generic (PLEG): container finished" podID="a568be4e-e6bb-4fd9-9328-be365f5fa0d6" containerID="2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c" exitCode=0 Aug 01 01:07:25.643906 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.643855 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" Aug 01 01:07:25.644124 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.643906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" event={"ID":"a568be4e-e6bb-4fd9-9328-be365f5fa0d6","Type":"ContainerDied","Data":"2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c"} Aug 01 01:07:25.644124 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.643948 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x" event={"ID":"a568be4e-e6bb-4fd9-9328-be365f5fa0d6","Type":"ContainerDied","Data":"0768a8a57c7da4b46b02f4a15f40a9714d60844416862a45c71da2dec884ec0c"} Aug 01 01:07:25.644124 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.643964 2575 scope.go:117] "RemoveContainer" containerID="2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c" Aug 01 01:07:25.653083 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.653065 2575 scope.go:117] "RemoveContainer" containerID="2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c" Aug 01 01:07:25.653408 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:07:25.653388 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c\": container with ID starting with 2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c not found: ID does not exist" containerID="2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c" Aug 01 01:07:25.653478 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.653420 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c"} err="failed to get container status \"2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c\": rpc error: code = NotFound desc = could not find container \"2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c\": container with ID starting with 2a55970076e7590c5187b108fd49d77151ad85b681872e480d13952113c0b18c not found: ID does not exist" Aug 01 01:07:25.664405 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.664361 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x"] Aug 01 01:07:25.671395 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:25.671369 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e23e15a8/e2e-lifecycle-model-e23e15a8-kserve-85f9896cf-ds67x"] Aug 01 01:07:27.163137 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:27.163095 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a568be4e-e6bb-4fd9-9328-be365f5fa0d6" path="/var/lib/kubelet/pods/a568be4e-e6bb-4fd9-9328-be365f5fa0d6/volumes" Aug 01 01:07:33.956126 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:33.956093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:34.011379 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:34.011349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:46.301898 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:46.301856 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:47.150433 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:47.150396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:52.769954 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:52.769908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:52.831802 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:52.831752 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:53.047453 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:53.047359 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:53.266589 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:53.266528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:07:54.869038 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:07:54.869004 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:08:04.520432 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:04.520388 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:04.619577 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:04.619543 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:04.719926 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:04.719893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:05.302365 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:05.302297 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:05.399078 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:05.399043 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:05.499443 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:05.499380 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:06.672969 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:06.672930 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:06.968323 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:06.968227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:08:12.470682 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:12.470642 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:12.566905 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:12.566875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:13.139982 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:13.139943 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:14.070811 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:14.070775 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:08:19.201627 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:19.201592 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:20.397304 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:20.397272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:29.871775 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:29.871738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:30.090899 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:30.090864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:51.031713 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:51.031676 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:56.872189 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:56.872128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:56.968600 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:56.968564 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:57.205491 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:57.205454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:57.493891 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:57.493805 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:58.039791 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:58.039758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:08:59.945328 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:08:59.945295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:00.045149 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:00.045117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:01.099850 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:01.099814 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:01.453556 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:01.453519 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:01.912576 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:01.912539 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:02.043736 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:02.043702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:04.559493 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:04.559455 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:09:11.269889 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:11.269862 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:09:11.272135 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:11.272114 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:09:24.931555 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:24.931494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:25.274838 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:25.274725 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:25.583550 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:25.583440 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:25.746733 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:25.746683 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:28.096216 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:28.096144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:28.199881 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:28.199839 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:29.303539 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:29.303503 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:29.831554 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:29.831512 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:29.930758 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:29.930726 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:30.015529 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:30.015491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:31.968711 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:31.968681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:09:40.579587 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:40.579503 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:09:53.054067 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:09:53.054031 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:10:20.243830 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.243797 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2"] Aug 01 01:10:20.246272 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.244110 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a568be4e-e6bb-4fd9-9328-be365f5fa0d6" containerName="main" Aug 01 01:10:20.246272 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.244122 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a568be4e-e6bb-4fd9-9328-be365f5fa0d6" containerName="main" Aug 01 01:10:20.246272 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.244204 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a568be4e-e6bb-4fd9-9328-be365f5fa0d6" containerName="main" Aug 01 01:10:20.247020 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.247004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.250809 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.250787 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-47a30d\"/\"test-model-47a30d-kserve-self-signed-certs\"" Aug 01 01:10:20.250967 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.250787 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-47a30d\"/\"kube-root-ca.crt\"" Aug 01 01:10:20.250967 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.250787 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-47a30d\"/\"openshift-service-ca.crt\"" Aug 01 01:10:20.250967 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.250812 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-47a30d\"/\"default-dockercfg-2l94l\"" Aug 01 01:10:20.257070 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.257038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2"] Aug 01 01:10:20.276958 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.276923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:10:20.301015 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.300978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5fcb\" (UniqueName: \"kubernetes.io/projected/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-kube-api-access-r5fcb\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.301215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.301023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tmp-dir\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.301215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.301048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-dshm\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.301215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.301070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tls-certs\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.301375 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.301212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-home\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.301375 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.301253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-model-cache\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402316 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5fcb\" (UniqueName: \"kubernetes.io/projected/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-kube-api-access-r5fcb\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402316 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tmp-dir\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402578 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-dshm\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402578 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tls-certs\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402578 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-home\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402578 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-model-cache\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402772 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402751 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tmp-dir\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402871 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-home\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.402871 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.402864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-model-cache\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.404696 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.404668 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-dshm\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.404904 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.404872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tls-certs\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.410268 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.410246 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5fcb\" (UniqueName: \"kubernetes.io/projected/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-kube-api-access-r5fcb\") pod \"test-model-47a30d-kserve-76fc5599d7-lbrj2\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.557620 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.557533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:20.693421 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:20.693389 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2"] Aug 01 01:10:20.694532 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:10:20.694509 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod855bc00f_3f0e_4ece_9922_4ca1f36fcd05.slice/crio-fe871f61b3a414faaccd557e76e9483b46acfe1816f0520d9e00bfe17ed65215 WatchSource:0}: Error finding container fe871f61b3a414faaccd557e76e9483b46acfe1816f0520d9e00bfe17ed65215: Status 404 returned error can't find the container with id fe871f61b3a414faaccd557e76e9483b46acfe1816f0520d9e00bfe17ed65215 Aug 01 01:10:21.490690 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:21.490657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" event={"ID":"855bc00f-3f0e-4ece-9922-4ca1f36fcd05","Type":"ContainerStarted","Data":"b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359"} Aug 01 01:10:21.490690 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:21.490700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" event={"ID":"855bc00f-3f0e-4ece-9922-4ca1f36fcd05","Type":"ContainerStarted","Data":"fe871f61b3a414faaccd557e76e9483b46acfe1816f0520d9e00bfe17ed65215"} Aug 01 01:10:21.507740 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:21.507687 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" podStartSLOduration=1.5076689760000002 podStartE2EDuration="1.507668976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:10:21.505740285 +0000 UTC m=+2470.862531933" watchObservedRunningTime="2026-08-01 01:10:21.507668976 +0000 UTC m=+2470.864460632" Aug 01 01:10:22.964648 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:22.964608 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:10:30.558145 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:30.558104 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:30.558692 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:30.558402 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:30.570872 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:30.570845 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:31.548865 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:31.548834 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:10:32.074887 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:32.074846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:10:55.745102 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.745063 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr"] Aug 01 01:10:55.748651 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.748633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:55.751041 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.751017 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-061b54\"/\"openshift-service-ca.crt\"" Aug 01 01:10:55.751167 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.751106 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-061b54\"/\"kube-root-ca.crt\"" Aug 01 01:10:55.752238 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.752223 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-061b54\"/\"default-dockercfg-fn598\"" Aug 01 01:10:55.752309 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.752288 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-061b54\"/\"test-model-061b54-kserve-self-signed-certs\"" Aug 01 01:10:55.755817 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.755791 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr"] Aug 01 01:10:55.774750 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.774704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:10:55.910282 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.910250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-dshm\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:55.910282 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.910284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-home\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:55.910521 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.910304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-model-cache\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:55.910521 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.910341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddv9c\" (UniqueName: \"kubernetes.io/projected/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-kube-api-access-ddv9c\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:55.910521 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.910376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tmp-dir\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:55.910521 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:55.910396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tls-certs\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.011627 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.011539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tmp-dir\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.011627 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.011576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tls-certs\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.011627 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.011610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-dshm\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.011873 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.011639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-home\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.011873 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.011666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-model-cache\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.011873 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.011715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv9c\" (UniqueName: \"kubernetes.io/projected/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-kube-api-access-ddv9c\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.012031 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.011992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tmp-dir\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.012109 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.012084 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-model-cache\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.012194 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.012125 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-home\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.014533 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.014510 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-dshm\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.014785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.014769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tls-certs\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.021457 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.021425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv9c\" (UniqueName: \"kubernetes.io/projected/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-kube-api-access-ddv9c\") pod \"test-model-061b54-kserve-6cdcff8877-vglmr\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.059837 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.059793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:10:56.189653 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.189632 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr"] Aug 01 01:10:56.191732 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:10:56.191706 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30874da4_a1cf_4c9f_bd55_199ecbbc57d8.slice/crio-d5689eefffaebdffe62a79aee1342ff82928535e83727fcc5e47fb73c8b39d7b WatchSource:0}: Error finding container d5689eefffaebdffe62a79aee1342ff82928535e83727fcc5e47fb73c8b39d7b: Status 404 returned error can't find the container with id d5689eefffaebdffe62a79aee1342ff82928535e83727fcc5e47fb73c8b39d7b Aug 01 01:10:56.655873 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.655841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" event={"ID":"30874da4-a1cf-4c9f-bd55-199ecbbc57d8","Type":"ContainerStarted","Data":"05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc"} Aug 01 01:10:56.656073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.655883 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" event={"ID":"30874da4-a1cf-4c9f-bd55-199ecbbc57d8","Type":"ContainerStarted","Data":"d5689eefffaebdffe62a79aee1342ff82928535e83727fcc5e47fb73c8b39d7b"} Aug 01 01:10:56.670787 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:56.670730 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" podStartSLOduration=1.670711602 podStartE2EDuration="1.670711602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:10:56.670217467 +0000 UTC m=+2506.027009125" watchObservedRunningTime="2026-08-01 01:10:56.670711602 +0000 UTC m=+2506.027503261" Aug 01 01:10:59.681153 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:10:59.681120 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:11:06.060173 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:06.060131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:11:06.060677 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:06.060223 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:11:06.072690 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:06.072664 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:11:06.716355 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:06.716328 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:11:43.426554 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:43.426515 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2"] Aug 01 01:11:43.453989 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:43.453943 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:11:44.288616 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:44.288577 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr"] Aug 01 01:11:44.301214 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:44.301128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:11:46.874767 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:46.874728 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:11:54.996742 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:54.996699 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:11:55.096441 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:55.096385 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:11:55.175101 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:55.175070 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:11:55.227385 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:55.227352 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:11:55.557863 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:55.557828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:11:57.169268 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:57.169229 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:11:58.445560 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.445493 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" podUID="855bc00f-3f0e-4ece-9922-4ca1f36fcd05" containerName="main" containerID="cri-o://b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359" gracePeriod=45 Aug 01 01:11:58.685794 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.685768 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:11:58.747848 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.747748 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-home\") pod \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " Aug 01 01:11:58.747848 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.747826 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-dshm\") pod \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " Aug 01 01:11:58.748088 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.747853 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5fcb\" (UniqueName: \"kubernetes.io/projected/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-kube-api-access-r5fcb\") pod \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " Aug 01 01:11:58.748088 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.747871 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-model-cache\") pod \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " Aug 01 01:11:58.748088 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.747923 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tmp-dir\") pod \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " Aug 01 01:11:58.748088 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.747970 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tls-certs\") pod \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\" (UID: \"855bc00f-3f0e-4ece-9922-4ca1f36fcd05\") " Aug 01 01:11:58.748088 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.748054 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-home" (OuterVolumeSpecName: "home") pod "855bc00f-3f0e-4ece-9922-4ca1f36fcd05" (UID: "855bc00f-3f0e-4ece-9922-4ca1f36fcd05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:58.748358 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.748206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-model-cache" (OuterVolumeSpecName: "model-cache") pod "855bc00f-3f0e-4ece-9922-4ca1f36fcd05" (UID: "855bc00f-3f0e-4ece-9922-4ca1f36fcd05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:58.748358 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.748227 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "855bc00f-3f0e-4ece-9922-4ca1f36fcd05" (UID: "855bc00f-3f0e-4ece-9922-4ca1f36fcd05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:58.748358 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.748271 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:58.750259 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.750237 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "855bc00f-3f0e-4ece-9922-4ca1f36fcd05" (UID: "855bc00f-3f0e-4ece-9922-4ca1f36fcd05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:11:58.750346 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.750323 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-kube-api-access-r5fcb" (OuterVolumeSpecName: "kube-api-access-r5fcb") pod "855bc00f-3f0e-4ece-9922-4ca1f36fcd05" (UID: "855bc00f-3f0e-4ece-9922-4ca1f36fcd05"). InnerVolumeSpecName "kube-api-access-r5fcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:11:58.750395 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.750340 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-dshm" (OuterVolumeSpecName: "dshm") pod "855bc00f-3f0e-4ece-9922-4ca1f36fcd05" (UID: "855bc00f-3f0e-4ece-9922-4ca1f36fcd05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:58.849097 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.849056 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:58.849097 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.849090 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:58.849097 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.849100 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:58.849378 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.849111 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5fcb\" (UniqueName: \"kubernetes.io/projected/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-kube-api-access-r5fcb\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:58.849378 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.849123 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855bc00f-3f0e-4ece-9922-4ca1f36fcd05-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:58.952573 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.952538 2575 generic.go:356] "Generic (PLEG): container finished" podID="855bc00f-3f0e-4ece-9922-4ca1f36fcd05" containerID="b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359" exitCode=0 Aug 01 01:11:58.952573 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.952576 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" Aug 01 01:11:58.952813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.952624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" event={"ID":"855bc00f-3f0e-4ece-9922-4ca1f36fcd05","Type":"ContainerDied","Data":"b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359"} Aug 01 01:11:58.952813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.952663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2" event={"ID":"855bc00f-3f0e-4ece-9922-4ca1f36fcd05","Type":"ContainerDied","Data":"fe871f61b3a414faaccd557e76e9483b46acfe1816f0520d9e00bfe17ed65215"} Aug 01 01:11:58.952813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.952680 2575 scope.go:117] "RemoveContainer" containerID="b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359" Aug 01 01:11:58.961450 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.961432 2575 scope.go:117] "RemoveContainer" containerID="b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359" Aug 01 01:11:58.961715 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:11:58.961699 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359\": container with ID starting with b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359 not found: ID does not exist" containerID="b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359" Aug 01 01:11:58.961774 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.961723 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359"} err="failed to get container status \"b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359\": rpc error: code = NotFound desc = could not find container \"b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359\": container with ID starting with b334f7ce7b17af584c75f5f55cd735d872d6acdc4d93136f4915c965a1017359 not found: ID does not exist" Aug 01 01:11:58.973507 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.973441 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2"] Aug 01 01:11:58.975252 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:58.975218 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-47a30d/test-model-47a30d-kserve-76fc5599d7-lbrj2"] Aug 01 01:11:59.163337 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.163307 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="855bc00f-3f0e-4ece-9922-4ca1f36fcd05" path="/var/lib/kubelet/pods/855bc00f-3f0e-4ece-9922-4ca1f36fcd05/volumes" Aug 01 01:11:59.308128 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.308071 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" podUID="30874da4-a1cf-4c9f-bd55-199ecbbc57d8" containerName="main" containerID="cri-o://05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc" gracePeriod=45 Aug 01 01:11:59.554386 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.554360 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:11:59.655033 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.654993 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tls-certs\") pod \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " Aug 01 01:11:59.655033 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655034 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-model-cache\") pod \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " Aug 01 01:11:59.655291 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-home\") pod \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " Aug 01 01:11:59.655291 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655073 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddv9c\" (UniqueName: \"kubernetes.io/projected/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-kube-api-access-ddv9c\") pod \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " Aug 01 01:11:59.655291 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655135 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-dshm\") pod \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " Aug 01 01:11:59.655291 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655174 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tmp-dir\") pod \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\" (UID: \"30874da4-a1cf-4c9f-bd55-199ecbbc57d8\") " Aug 01 01:11:59.655499 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655336 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-home" (OuterVolumeSpecName: "home") pod "30874da4-a1cf-4c9f-bd55-199ecbbc57d8" (UID: "30874da4-a1cf-4c9f-bd55-199ecbbc57d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:59.655499 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655353 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "30874da4-a1cf-4c9f-bd55-199ecbbc57d8" (UID: "30874da4-a1cf-4c9f-bd55-199ecbbc57d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:59.655584 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.655498 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30874da4-a1cf-4c9f-bd55-199ecbbc57d8" (UID: "30874da4-a1cf-4c9f-bd55-199ecbbc57d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:59.657384 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.657351 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-dshm" (OuterVolumeSpecName: "dshm") pod "30874da4-a1cf-4c9f-bd55-199ecbbc57d8" (UID: "30874da4-a1cf-4c9f-bd55-199ecbbc57d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:11:59.657384 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.657373 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30874da4-a1cf-4c9f-bd55-199ecbbc57d8" (UID: "30874da4-a1cf-4c9f-bd55-199ecbbc57d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:11:59.657529 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.657416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-kube-api-access-ddv9c" (OuterVolumeSpecName: "kube-api-access-ddv9c") pod "30874da4-a1cf-4c9f-bd55-199ecbbc57d8" (UID: "30874da4-a1cf-4c9f-bd55-199ecbbc57d8"). InnerVolumeSpecName "kube-api-access-ddv9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:11:59.756064 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.755969 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:59.756064 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.756003 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:59.756064 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.756012 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddv9c\" (UniqueName: \"kubernetes.io/projected/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-kube-api-access-ddv9c\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:59.756064 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.756020 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:59.756064 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.756028 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:59.756064 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.756037 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30874da4-a1cf-4c9f-bd55-199ecbbc57d8-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:11:59.958790 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.958751 2575 generic.go:356] "Generic (PLEG): container finished" podID="30874da4-a1cf-4c9f-bd55-199ecbbc57d8" containerID="05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc" exitCode=0 Aug 01 01:11:59.958963 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.958798 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" Aug 01 01:11:59.958963 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.958839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" event={"ID":"30874da4-a1cf-4c9f-bd55-199ecbbc57d8","Type":"ContainerDied","Data":"05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc"} Aug 01 01:11:59.958963 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.958883 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr" event={"ID":"30874da4-a1cf-4c9f-bd55-199ecbbc57d8","Type":"ContainerDied","Data":"d5689eefffaebdffe62a79aee1342ff82928535e83727fcc5e47fb73c8b39d7b"} Aug 01 01:11:59.958963 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.958905 2575 scope.go:117] "RemoveContainer" containerID="05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc" Aug 01 01:11:59.967857 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.967840 2575 scope.go:117] "RemoveContainer" containerID="05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc" Aug 01 01:11:59.968142 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:11:59.968121 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc\": container with ID starting with 05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc not found: ID does not exist" containerID="05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc" Aug 01 01:11:59.968206 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.968152 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc"} err="failed to get container status \"05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc\": rpc error: code = NotFound desc = could not find container \"05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc\": container with ID starting with 05318503d6d4d45db6c9e98fd9a5401062657975bf5a683d2fd2bb4a38877afc not found: ID does not exist" Aug 01 01:11:59.979260 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.979233 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr"] Aug 01 01:11:59.983210 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:11:59.983187 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-061b54/test-model-061b54-kserve-6cdcff8877-vglmr"] Aug 01 01:12:01.163430 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:01.163393 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30874da4-a1cf-4c9f-bd55-199ecbbc57d8" path="/var/lib/kubelet/pods/30874da4-a1cf-4c9f-bd55-199ecbbc57d8/volumes" Aug 01 01:12:01.612372 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:01.612282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:01.713033 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:01.712994 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:01.867403 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:01.867321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:02.074509 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:02.074475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:06.375858 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:06.375818 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:12.632823 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:12.632788 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:12.733777 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:12.733715 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:12.810755 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:12.810722 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:12.872919 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:12.872883 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:13.225654 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:13.225622 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:14.868151 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:14.868113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:12:19.035335 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:19.035300 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:19.139709 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:19.139672 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:19.293390 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:19.293305 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:19.504859 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:19.504830 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:19.857443 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:19.857392 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:20.148575 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:20.148490 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:20.444148 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:20.444112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:23.012071 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:23.012034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:23.111686 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:23.111643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:24.139820 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:24.139756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:24.437147 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:24.437112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:24.633430 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:24.633387 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:24.719841 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:24.719756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:28.667921 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:28.667885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:12:47.959810 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:47.959762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:48.417791 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:48.417755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:48.700044 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:48.699941 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:48.883440 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:48.883396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:51.247749 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:51.247716 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:51.347540 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:51.347506 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:52.366420 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:52.366389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:52.524440 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:52.524411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:52.870490 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:52.870455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:52.961305 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:52.961266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:12:55.762024 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:55.761981 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:12:57.034582 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:12:57.034528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:16.129902 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:16.129859 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:34.415305 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:34.415265 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:35.522862 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:35.522826 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:36.764226 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:36.764186 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:13:40.509911 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:40.509809 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:40.567762 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:40.567723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:40.943606 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:40.943563 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:41.460262 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:41.460216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:52.687804 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:52.687767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:57.251038 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:57.251005 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:57.350876 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:57.350823 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:57.412359 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:57.412327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:57.465018 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:57.464987 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:57.783669 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:57.783631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:13:59.170557 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:13:59.170520 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:14:03.810484 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:03.810418 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:03.879097 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:03.879056 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:04.062370 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:04.062269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:04.286495 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:04.286460 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:04.617015 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:04.616984 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:04.911294 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:04.911263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:05.368302 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:05.368200 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:07.642959 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:07.642926 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:07.743271 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:07.743239 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:08.768766 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:08.768723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:08.878452 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:08.878401 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:09.286684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:09.286651 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:09.383025 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:09.382991 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:11.299692 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:11.299665 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:14:11.302122 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:11.302103 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:14:11.469480 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:11.469438 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:14:21.388744 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:21.388703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:21.450087 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:21.450052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:32.533095 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:32.533065 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:32.853744 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:32.853643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:33.191588 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:33.191559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:33.424632 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:33.424600 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:35.739795 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:35.739760 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:35.836084 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:35.836048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:36.858178 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:36.858135 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:37.216496 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:37.216463 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:37.340486 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:37.340451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:37.438812 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:37.438775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:14:39.985490 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:14:39.985452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:15:00.698951 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:00.698913 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:15:10.839672 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.839636 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865"] Aug 01 01:15:10.840194 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.840051 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30874da4-a1cf-4c9f-bd55-199ecbbc57d8" containerName="main" Aug 01 01:15:10.840194 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.840068 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="30874da4-a1cf-4c9f-bd55-199ecbbc57d8" containerName="main" Aug 01 01:15:10.840194 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.840091 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855bc00f-3f0e-4ece-9922-4ca1f36fcd05" containerName="main" Aug 01 01:15:10.840194 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.840098 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="855bc00f-3f0e-4ece-9922-4ca1f36fcd05" containerName="main" Aug 01 01:15:10.840194 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.840196 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="855bc00f-3f0e-4ece-9922-4ca1f36fcd05" containerName="main" Aug 01 01:15:10.840468 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.840210 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="30874da4-a1cf-4c9f-bd55-199ecbbc57d8" containerName="main" Aug 01 01:15:10.843201 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.843177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:10.845741 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.845719 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1e2f5b\"/\"kube-root-ca.crt\"" Aug 01 01:15:10.846823 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.846792 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1e2f5b\"/\"openshift-service-ca.crt\"" Aug 01 01:15:10.846956 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.846938 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e2f5b\"/\"auth-test-model-1e2f5b-kserve-self-signed-certs\"" Aug 01 01:15:10.847067 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.847026 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e2f5b\"/\"default-dockercfg-55b56\"" Aug 01 01:15:10.854062 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.854038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865"] Aug 01 01:15:10.872293 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.872243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:15:10.918546 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.918508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b334e308-2c6e-4536-902d-b2f973832758-tls-certs\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:10.918546 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.918548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-home\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:10.918791 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.918573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-model-cache\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:10.918791 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.918590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-dshm\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:10.918791 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.918671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcwk\" (UniqueName: \"kubernetes.io/projected/b334e308-2c6e-4536-902d-b2f973832758-kube-api-access-mzcwk\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:10.918791 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:10.918709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-tmp-dir\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019420 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcwk\" (UniqueName: \"kubernetes.io/projected/b334e308-2c6e-4536-902d-b2f973832758-kube-api-access-mzcwk\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019420 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-tmp-dir\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019804 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b334e308-2c6e-4536-902d-b2f973832758-tls-certs\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019804 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-home\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019804 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-model-cache\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019804 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-dshm\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019948 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-tmp-dir\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.019948 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-home\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.020012 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.019994 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-model-cache\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.021849 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.021827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-dshm\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.022170 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.022140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b334e308-2c6e-4536-902d-b2f973832758-tls-certs\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.028735 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.028712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcwk\" (UniqueName: \"kubernetes.io/projected/b334e308-2c6e-4536-902d-b2f973832758-kube-api-access-mzcwk\") pod \"auth-test-model-1e2f5b-kserve-65f6b4454c-9t865\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.154710 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.154681 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e2f5b\"/\"default-dockercfg-55b56\"" Aug 01 01:15:11.162845 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.162817 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:11.295287 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.294571 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865"] Aug 01 01:15:11.868422 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.868378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" event={"ID":"b334e308-2c6e-4536-902d-b2f973832758","Type":"ContainerStarted","Data":"b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13"} Aug 01 01:15:11.868422 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.868421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" event={"ID":"b334e308-2c6e-4536-902d-b2f973832758","Type":"ContainerStarted","Data":"9225cb896145db653da7830f5808a78bda116b0cd2d63f46809226cf0d02e0c4"} Aug 01 01:15:11.883779 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:11.883723 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" podStartSLOduration=1.883706719 podStartE2EDuration="1.883706719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:15:11.882988239 +0000 UTC m=+2761.239779894" watchObservedRunningTime="2026-08-01 01:15:11.883706719 +0000 UTC m=+2761.240498376" Aug 01 01:15:13.072635 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:13.072597 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:15:21.163946 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:21.163916 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:21.163946 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:21.163956 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:21.176336 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:21.176314 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:21.924260 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:21.924229 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:15:44.939358 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.939324 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2"] Aug 01 01:15:44.942739 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.942718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:44.945395 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.945371 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"kube-root-ca.crt\"" Aug 01 01:15:44.946422 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.946394 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"auth-test-model-4224e1-kserve-self-signed-certs\"" Aug 01 01:15:44.946533 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.946416 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"default-dockercfg-qpp6c\"" Aug 01 01:15:44.946533 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.946493 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"openshift-service-ca.crt\"" Aug 01 01:15:44.952575 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.952554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2"] Aug 01 01:15:44.976827 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:44.976802 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:15:45.117600 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.117567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrp8w\" (UniqueName: \"kubernetes.io/projected/4aed2b66-94df-4753-8dfe-b56c7c13cee2-kube-api-access-jrp8w\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.117600 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.117607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-home\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.117828 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.117630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-model-cache\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.117828 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.117699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tmp-dir\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.117828 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.117748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-dshm\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.117828 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.117771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tls-certs\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218267 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tmp-dir\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218267 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-dshm\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218502 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tls-certs\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218502 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrp8w\" (UniqueName: \"kubernetes.io/projected/4aed2b66-94df-4753-8dfe-b56c7c13cee2-kube-api-access-jrp8w\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218502 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-home\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218502 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-model-cache\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218711 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tmp-dir\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218711 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-model-cache\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.218792 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.218768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-home\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.220536 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.220512 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-dshm\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.220797 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.220778 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tls-certs\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.225918 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.225895 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrp8w\" (UniqueName: \"kubernetes.io/projected/4aed2b66-94df-4753-8dfe-b56c7c13cee2-kube-api-access-jrp8w\") pod \"auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.253950 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.253916 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:45.383640 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:45.383610 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2"] Aug 01 01:15:45.385789 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:15:45.385766 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4aed2b66_94df_4753_8dfe_b56c7c13cee2.slice/crio-186b3c1c92773a8868319cc4d7d92cb77137817c9bbab21151ea466ee3be42a1 WatchSource:0}: Error finding container 186b3c1c92773a8868319cc4d7d92cb77137817c9bbab21151ea466ee3be42a1: Status 404 returned error can't find the container with id 186b3c1c92773a8868319cc4d7d92cb77137817c9bbab21151ea466ee3be42a1 Aug 01 01:15:46.020997 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:46.020959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" event={"ID":"4aed2b66-94df-4753-8dfe-b56c7c13cee2","Type":"ContainerStarted","Data":"9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509"} Aug 01 01:15:46.020997 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:46.020999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" event={"ID":"4aed2b66-94df-4753-8dfe-b56c7c13cee2","Type":"ContainerStarted","Data":"186b3c1c92773a8868319cc4d7d92cb77137817c9bbab21151ea466ee3be42a1"} Aug 01 01:15:46.038398 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:46.038347 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" podStartSLOduration=2.038333574 podStartE2EDuration="2.038333574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:15:46.036034788 +0000 UTC m=+2795.392826444" watchObservedRunningTime="2026-08-01 01:15:46.038333574 +0000 UTC m=+2795.395125230" Aug 01 01:15:47.462999 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:47.462961 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:15:55.254011 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:55.253974 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:55.254460 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:55.254031 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:55.266320 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:55.266301 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:15:56.076829 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:15:56.076799 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:16:11.062669 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:16:11.062631 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:16:26.574332 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:16:26.574295 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:16:37.676576 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:16:37.676539 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:16:53.263469 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:16:53.263427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:17:03.963355 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:17:03.963319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:17:19.169102 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:17:19.169070 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:17:29.773250 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:17:29.773210 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:17:36.963491 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:17:36.963453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:17:50.861728 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:17:50.861678 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:18:02.169917 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:02.169876 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:18:17.065509 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:17.065472 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:18:28.568228 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:28.568185 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:18:43.273189 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:43.273087 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:18:47.370215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:47.370151 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865"] Aug 01 01:18:47.424136 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:47.424101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:18:48.146190 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:48.146135 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2"] Aug 01 01:18:48.495993 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:48.495953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:18:53.273813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:53.273777 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:18:59.764115 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.764080 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc"] Aug 01 01:18:59.766873 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.766842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:18:59.767386 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.767368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.769755 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.769738 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e2f5b\"/\"sub-test-model-1e2f5b-kserve-self-signed-certs\"" Aug 01 01:18:59.776870 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.776843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc"] Aug 01 01:18:59.859758 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.859715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-model-cache\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.859758 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.859762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbns\" (UniqueName: \"kubernetes.io/projected/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-kube-api-access-dwbns\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.859994 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.859799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tmp-dir\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.859994 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.859850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tls-certs\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.859994 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.859886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-dshm\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.859994 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.859908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-home\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961361 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tmp-dir\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961361 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tls-certs\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961629 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-dshm\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961629 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-home\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961629 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-model-cache\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961629 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbns\" (UniqueName: \"kubernetes.io/projected/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-kube-api-access-dwbns\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961856 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tmp-dir\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961914 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961895 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-model-cache\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.961951 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.961892 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-home\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.963749 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.963726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-dshm\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.963975 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.963956 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tls-certs\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:18:59.974669 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:18:59.971063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbns\" (UniqueName: \"kubernetes.io/projected/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-kube-api-access-dwbns\") pod \"sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:19:00.076891 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:00.076796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:19:00.210677 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:00.210650 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc"] Aug 01 01:19:00.212269 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:19:00.212244 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2557e078_6aa3_4cb4_9fb7_e708d461e1c2.slice/crio-5af323db5921edaaac13e904498bdd8601e24466869cf9f739b9ca8eaa73ebc4 WatchSource:0}: Error finding container 5af323db5921edaaac13e904498bdd8601e24466869cf9f739b9ca8eaa73ebc4: Status 404 returned error can't find the container with id 5af323db5921edaaac13e904498bdd8601e24466869cf9f739b9ca8eaa73ebc4 Aug 01 01:19:00.929882 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:00.929843 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" event={"ID":"2557e078-6aa3-4cb4-9fb7-e708d461e1c2","Type":"ContainerStarted","Data":"6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3"} Aug 01 01:19:00.930325 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:00.929891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" event={"ID":"2557e078-6aa3-4cb4-9fb7-e708d461e1c2","Type":"ContainerStarted","Data":"5af323db5921edaaac13e904498bdd8601e24466869cf9f739b9ca8eaa73ebc4"} Aug 01 01:19:00.948258 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:00.948203 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" podStartSLOduration=1.94818568 podStartE2EDuration="1.94818568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:19:00.946470832 +0000 UTC m=+2990.303262488" watchObservedRunningTime="2026-08-01 01:19:00.94818568 +0000 UTC m=+2990.304977335" Aug 01 01:19:02.074122 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.074086 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:19:02.389425 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.389337 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" podUID="b334e308-2c6e-4536-902d-b2f973832758" containerName="main" containerID="cri-o://b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13" gracePeriod=45 Aug 01 01:19:02.630889 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.630859 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:19:02.685821 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.685773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b334e308-2c6e-4536-902d-b2f973832758-tls-certs\") pod \"b334e308-2c6e-4536-902d-b2f973832758\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " Aug 01 01:19:02.685821 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.685829 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-tmp-dir\") pod \"b334e308-2c6e-4536-902d-b2f973832758\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " Aug 01 01:19:02.686087 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.685858 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-home\") pod \"b334e308-2c6e-4536-902d-b2f973832758\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " Aug 01 01:19:02.686087 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.685886 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzcwk\" (UniqueName: \"kubernetes.io/projected/b334e308-2c6e-4536-902d-b2f973832758-kube-api-access-mzcwk\") pod \"b334e308-2c6e-4536-902d-b2f973832758\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " Aug 01 01:19:02.686087 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686057 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-dshm\") pod \"b334e308-2c6e-4536-902d-b2f973832758\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " Aug 01 01:19:02.686273 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686115 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-model-cache\") pod \"b334e308-2c6e-4536-902d-b2f973832758\" (UID: \"b334e308-2c6e-4536-902d-b2f973832758\") " Aug 01 01:19:02.686273 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686128 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-home" (OuterVolumeSpecName: "home") pod "b334e308-2c6e-4536-902d-b2f973832758" (UID: "b334e308-2c6e-4536-902d-b2f973832758"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:02.686273 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686144 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b334e308-2c6e-4536-902d-b2f973832758" (UID: "b334e308-2c6e-4536-902d-b2f973832758"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:02.686420 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686334 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-model-cache" (OuterVolumeSpecName: "model-cache") pod "b334e308-2c6e-4536-902d-b2f973832758" (UID: "b334e308-2c6e-4536-902d-b2f973832758"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:02.686476 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686429 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:02.686476 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686449 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:02.686476 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.686462 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:02.688250 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.688224 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-dshm" (OuterVolumeSpecName: "dshm") pod "b334e308-2c6e-4536-902d-b2f973832758" (UID: "b334e308-2c6e-4536-902d-b2f973832758"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:02.688625 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.688606 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b334e308-2c6e-4536-902d-b2f973832758-kube-api-access-mzcwk" (OuterVolumeSpecName: "kube-api-access-mzcwk") pod "b334e308-2c6e-4536-902d-b2f973832758" (UID: "b334e308-2c6e-4536-902d-b2f973832758"). InnerVolumeSpecName "kube-api-access-mzcwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:19:02.688689 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.688618 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b334e308-2c6e-4536-902d-b2f973832758-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b334e308-2c6e-4536-902d-b2f973832758" (UID: "b334e308-2c6e-4536-902d-b2f973832758"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:19:02.787374 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.787345 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcwk\" (UniqueName: \"kubernetes.io/projected/b334e308-2c6e-4536-902d-b2f973832758-kube-api-access-mzcwk\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:02.787374 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.787370 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b334e308-2c6e-4536-902d-b2f973832758-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:02.787374 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.787378 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b334e308-2c6e-4536-902d-b2f973832758-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:02.939937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.939847 2575 generic.go:356] "Generic (PLEG): container finished" podID="b334e308-2c6e-4536-902d-b2f973832758" containerID="b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13" exitCode=0 Aug 01 01:19:02.939937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.939891 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" Aug 01 01:19:02.939937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.939907 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" event={"ID":"b334e308-2c6e-4536-902d-b2f973832758","Type":"ContainerDied","Data":"b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13"} Aug 01 01:19:02.939937 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.939938 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865" event={"ID":"b334e308-2c6e-4536-902d-b2f973832758","Type":"ContainerDied","Data":"9225cb896145db653da7830f5808a78bda116b0cd2d63f46809226cf0d02e0c4"} Aug 01 01:19:02.940221 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.939955 2575 scope.go:117] "RemoveContainer" containerID="b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13" Aug 01 01:19:02.949030 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.949010 2575 scope.go:117] "RemoveContainer" containerID="b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13" Aug 01 01:19:02.949338 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:19:02.949319 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13\": container with ID starting with b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13 not found: ID does not exist" containerID="b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13" Aug 01 01:19:02.949396 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.949347 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13"} err="failed to get container status \"b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13\": rpc error: code = NotFound desc = could not find container \"b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13\": container with ID starting with b93f8a7ff3717772c13dbc555bde8808da0199b4e6dfba05a712607a71e16f13 not found: ID does not exist" Aug 01 01:19:02.960838 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.960807 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865"] Aug 01 01:19:02.964681 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:02.964654 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/auth-test-model-1e2f5b-kserve-65f6b4454c-9t865"] Aug 01 01:19:03.163522 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.163486 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b334e308-2c6e-4536-902d-b2f973832758" path="/var/lib/kubelet/pods/b334e308-2c6e-4536-902d-b2f973832758/volumes" Aug 01 01:19:03.165596 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.165567 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" podUID="4aed2b66-94df-4753-8dfe-b56c7c13cee2" containerName="main" containerID="cri-o://9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509" gracePeriod=45 Aug 01 01:19:03.397756 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.397733 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:19:03.494575 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494540 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-model-cache\") pod \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " Aug 01 01:19:03.494785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494601 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tmp-dir\") pod \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " Aug 01 01:19:03.494785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-dshm\") pod \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " Aug 01 01:19:03.494785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494651 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrp8w\" (UniqueName: \"kubernetes.io/projected/4aed2b66-94df-4753-8dfe-b56c7c13cee2-kube-api-access-jrp8w\") pod \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " Aug 01 01:19:03.494785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494709 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tls-certs\") pod \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " Aug 01 01:19:03.494785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494746 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-home\") pod \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\" (UID: \"4aed2b66-94df-4753-8dfe-b56c7c13cee2\") " Aug 01 01:19:03.495002 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494849 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4aed2b66-94df-4753-8dfe-b56c7c13cee2" (UID: "4aed2b66-94df-4753-8dfe-b56c7c13cee2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:03.495002 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.494901 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4aed2b66-94df-4753-8dfe-b56c7c13cee2" (UID: "4aed2b66-94df-4753-8dfe-b56c7c13cee2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:03.495107 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.495006 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:03.495107 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.495050 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:03.495107 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.495070 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-home" (OuterVolumeSpecName: "home") pod "4aed2b66-94df-4753-8dfe-b56c7c13cee2" (UID: "4aed2b66-94df-4753-8dfe-b56c7c13cee2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:03.496976 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.496953 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-dshm" (OuterVolumeSpecName: "dshm") pod "4aed2b66-94df-4753-8dfe-b56c7c13cee2" (UID: "4aed2b66-94df-4753-8dfe-b56c7c13cee2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:19:03.497054 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.497033 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4aed2b66-94df-4753-8dfe-b56c7c13cee2" (UID: "4aed2b66-94df-4753-8dfe-b56c7c13cee2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:19:03.497094 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.497068 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aed2b66-94df-4753-8dfe-b56c7c13cee2-kube-api-access-jrp8w" (OuterVolumeSpecName: "kube-api-access-jrp8w") pod "4aed2b66-94df-4753-8dfe-b56c7c13cee2" (UID: "4aed2b66-94df-4753-8dfe-b56c7c13cee2"). InnerVolumeSpecName "kube-api-access-jrp8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:19:03.596455 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.596362 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed2b66-94df-4753-8dfe-b56c7c13cee2-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:03.596455 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.596392 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:03.596455 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.596400 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed2b66-94df-4753-8dfe-b56c7c13cee2-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:03.596455 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.596408 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrp8w\" (UniqueName: \"kubernetes.io/projected/4aed2b66-94df-4753-8dfe-b56c7c13cee2-kube-api-access-jrp8w\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:19:03.947137 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.947101 2575 generic.go:356] "Generic (PLEG): container finished" podID="4aed2b66-94df-4753-8dfe-b56c7c13cee2" containerID="9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509" exitCode=0 Aug 01 01:19:03.947310 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.947149 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" Aug 01 01:19:03.947310 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.947198 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" event={"ID":"4aed2b66-94df-4753-8dfe-b56c7c13cee2","Type":"ContainerDied","Data":"9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509"} Aug 01 01:19:03.947310 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.947245 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2" event={"ID":"4aed2b66-94df-4753-8dfe-b56c7c13cee2","Type":"ContainerDied","Data":"186b3c1c92773a8868319cc4d7d92cb77137817c9bbab21151ea466ee3be42a1"} Aug 01 01:19:03.947310 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.947264 2575 scope.go:117] "RemoveContainer" containerID="9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509" Aug 01 01:19:03.956434 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.956408 2575 scope.go:117] "RemoveContainer" containerID="9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509" Aug 01 01:19:03.956721 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:19:03.956700 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509\": container with ID starting with 9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509 not found: ID does not exist" containerID="9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509" Aug 01 01:19:03.956775 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.956730 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509"} err="failed to get container status \"9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509\": rpc error: code = NotFound desc = could not find container \"9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509\": container with ID starting with 9af9fd4caef2735547d9779aa0fd6f36b8b8e44bd2ef54135e7b8bb836d79509 not found: ID does not exist" Aug 01 01:19:03.967263 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.967232 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2"] Aug 01 01:19:03.972920 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:03.972893 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/auth-test-model-4224e1-kserve-7c45f4c5d7-6knw2"] Aug 01 01:19:05.162854 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:05.162814 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aed2b66-94df-4753-8dfe-b56c7c13cee2" path="/var/lib/kubelet/pods/4aed2b66-94df-4753-8dfe-b56c7c13cee2/volumes" Aug 01 01:19:10.077434 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:10.077396 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:19:10.077849 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:10.077449 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:19:10.090145 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:10.090118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:19:10.993150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:10.993118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:19:11.328899 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:11.328819 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:19:11.332392 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:11.332373 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:19:19.742013 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.741974 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt"] Aug 01 01:19:19.742531 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.742304 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b334e308-2c6e-4536-902d-b2f973832758" containerName="main" Aug 01 01:19:19.742531 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.742316 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b334e308-2c6e-4536-902d-b2f973832758" containerName="main" Aug 01 01:19:19.742531 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.742326 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed2b66-94df-4753-8dfe-b56c7c13cee2" containerName="main" Aug 01 01:19:19.742531 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.742331 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed2b66-94df-4753-8dfe-b56c7c13cee2" containerName="main" Aug 01 01:19:19.742531 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.742395 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aed2b66-94df-4753-8dfe-b56c7c13cee2" containerName="main" Aug 01 01:19:19.742531 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.742417 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b334e308-2c6e-4536-902d-b2f973832758" containerName="main" Aug 01 01:19:19.746707 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.746687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.750214 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.750142 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"openshift-service-ca.crt\"" Aug 01 01:19:19.750214 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.750195 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"default-dockercfg-qpp6c\"" Aug 01 01:19:19.750450 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.750231 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"kube-root-ca.crt\"" Aug 01 01:19:19.750450 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.750273 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"sub-test-model-4224e1-kserve-self-signed-certs\"" Aug 01 01:19:19.753785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.753756 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt"] Aug 01 01:19:19.765320 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.765283 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:19:19.832572 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.832523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-dshm\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.832572 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.832583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865vl\" (UniqueName: \"kubernetes.io/projected/d803ee27-2fb0-48c5-92f1-279c2326284d-kube-api-access-865vl\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.832795 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.832655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-home\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.832795 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.832703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d803ee27-2fb0-48c5-92f1-279c2326284d-tls-certs\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.832795 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.832723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-tmp-dir\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.832795 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.832749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-model-cache\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.933770 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.933734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-dshm\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.933939 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.933784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865vl\" (UniqueName: \"kubernetes.io/projected/d803ee27-2fb0-48c5-92f1-279c2326284d-kube-api-access-865vl\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.933939 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.933900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-home\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.934020 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.933961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d803ee27-2fb0-48c5-92f1-279c2326284d-tls-certs\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.934020 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.933984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-tmp-dir\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.934141 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.934023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-model-cache\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.934296 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.934276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-home\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.934362 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.934331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-model-cache\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.934420 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.934405 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-tmp-dir\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.936085 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.936057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-dshm\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.936397 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.936379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d803ee27-2fb0-48c5-92f1-279c2326284d-tls-certs\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:19.941469 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:19.941445 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-865vl\" (UniqueName: \"kubernetes.io/projected/d803ee27-2fb0-48c5-92f1-279c2326284d-kube-api-access-865vl\") pod \"sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:20.057233 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:20.057112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:20.192381 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:20.192349 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt"] Aug 01 01:19:20.194221 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:19:20.194192 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd803ee27_2fb0_48c5_92f1_279c2326284d.slice/crio-b4437020414b67226c4358a77f48fd7502989a89cc856b225fe7b99ac7ca67aa WatchSource:0}: Error finding container b4437020414b67226c4358a77f48fd7502989a89cc856b225fe7b99ac7ca67aa: Status 404 returned error can't find the container with id b4437020414b67226c4358a77f48fd7502989a89cc856b225fe7b99ac7ca67aa Aug 01 01:19:21.031062 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:21.031023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" event={"ID":"d803ee27-2fb0-48c5-92f1-279c2326284d","Type":"ContainerStarted","Data":"dc1d9e550cd8d8b972d9b4e2547fb7cc299003123caf133409059531dd1cacdd"} Aug 01 01:19:21.031062 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:21.031067 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" event={"ID":"d803ee27-2fb0-48c5-92f1-279c2326284d","Type":"ContainerStarted","Data":"b4437020414b67226c4358a77f48fd7502989a89cc856b225fe7b99ac7ca67aa"} Aug 01 01:19:21.049206 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:21.049132 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" podStartSLOduration=2.049115987 podStartE2EDuration="2.049115987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:19:21.047489988 +0000 UTC m=+3010.404281640" watchObservedRunningTime="2026-08-01 01:19:21.049115987 +0000 UTC m=+3010.405907643" Aug 01 01:19:22.272816 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:22.272779 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:19:30.057500 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:30.057460 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:30.057978 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:30.057674 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:30.069995 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:30.069961 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:30.086003 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:30.085976 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:19:55.378069 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:19:55.378030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:20:03.243906 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:03.243867 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:20:07.773104 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:07.773063 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:20:20.064347 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:20.064313 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:20:25.346087 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:25.346051 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc"] Aug 01 01:20:25.366770 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:25.366737 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:20:26.225581 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:26.225550 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt"] Aug 01 01:20:26.246202 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:26.246168 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:20:28.369355 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:28.369318 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:20:37.542764 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.542720 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf"] Aug 01 01:20:37.546401 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.546383 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.548753 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.548731 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1e2f5b\"/\"rate-test-model-1e2f5b-kserve-self-signed-certs\"" Aug 01 01:20:37.553258 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.553093 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf"] Aug 01 01:20:37.569550 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.569520 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:20:37.694376 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.694335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-dshm\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.694576 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.694385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dda91f-a228-45e0-a376-082b90101630-tls-certs\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.694576 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.694479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-model-cache\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.694576 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.694526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2blx\" (UniqueName: \"kubernetes.io/projected/e4dda91f-a228-45e0-a376-082b90101630-kube-api-access-w2blx\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.694576 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.694563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-tmp-dir\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.694726 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.694587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-home\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796023 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.795935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2blx\" (UniqueName: \"kubernetes.io/projected/e4dda91f-a228-45e0-a376-082b90101630-kube-api-access-w2blx\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796023 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.795976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-tmp-dir\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796023 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.795996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-home\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796276 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.796044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-dshm\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796276 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.796077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dda91f-a228-45e0-a376-082b90101630-tls-certs\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796276 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.796125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-model-cache\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796452 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.796430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-tmp-dir\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796495 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.796473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-home\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.796555 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.796540 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-model-cache\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.798491 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.798467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-dshm\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.798577 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.798549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dda91f-a228-45e0-a376-082b90101630-tls-certs\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.804338 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.804318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2blx\" (UniqueName: \"kubernetes.io/projected/e4dda91f-a228-45e0-a376-082b90101630-kube-api-access-w2blx\") pod \"rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.856044 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.856006 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:37.987654 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:37.987626 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf"] Aug 01 01:20:37.989652 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:20:37.989624 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4dda91f_a228_45e0_a376_082b90101630.slice/crio-c2121fdac7fc9ca1328532acebe97b070c111bd937146968602f6180c5123cda WatchSource:0}: Error finding container c2121fdac7fc9ca1328532acebe97b070c111bd937146968602f6180c5123cda: Status 404 returned error can't find the container with id c2121fdac7fc9ca1328532acebe97b070c111bd937146968602f6180c5123cda Aug 01 01:20:38.422274 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:38.422238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" event={"ID":"e4dda91f-a228-45e0-a376-082b90101630","Type":"ContainerStarted","Data":"485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d"} Aug 01 01:20:38.422274 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:38.422279 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" event={"ID":"e4dda91f-a228-45e0-a376-082b90101630","Type":"ContainerStarted","Data":"c2121fdac7fc9ca1328532acebe97b070c111bd937146968602f6180c5123cda"} Aug 01 01:20:38.441010 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:38.440951 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" podStartSLOduration=1.440933769 podStartE2EDuration="1.440933769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:20:38.439254113 +0000 UTC m=+3087.796045768" watchObservedRunningTime="2026-08-01 01:20:38.440933769 +0000 UTC m=+3087.797725431" Aug 01 01:20:39.765133 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:39.765095 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:20:40.365238 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.365176 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" podUID="2557e078-6aa3-4cb4-9fb7-e708d461e1c2" containerName="main" containerID="cri-o://6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3" gracePeriod=45 Aug 01 01:20:40.622534 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.622465 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:20:40.723626 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723580 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwbns\" (UniqueName: \"kubernetes.io/projected/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-kube-api-access-dwbns\") pod \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " Aug 01 01:20:40.723626 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723627 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-home\") pod \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " Aug 01 01:20:40.723876 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723653 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tmp-dir\") pod \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " Aug 01 01:20:40.723876 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723797 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tls-certs\") pod \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " Aug 01 01:20:40.723876 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723840 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-model-cache\") pod \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " Aug 01 01:20:40.723876 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723860 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-dshm\") pod \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\" (UID: \"2557e078-6aa3-4cb4-9fb7-e708d461e1c2\") " Aug 01 01:20:40.724073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723908 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2557e078-6aa3-4cb4-9fb7-e708d461e1c2" (UID: "2557e078-6aa3-4cb4-9fb7-e708d461e1c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:40.724073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.723966 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-home" (OuterVolumeSpecName: "home") pod "2557e078-6aa3-4cb4-9fb7-e708d461e1c2" (UID: "2557e078-6aa3-4cb4-9fb7-e708d461e1c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:40.724144 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.724109 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2557e078-6aa3-4cb4-9fb7-e708d461e1c2" (UID: "2557e078-6aa3-4cb4-9fb7-e708d461e1c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:40.724217 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.724183 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:40.724217 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.724198 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:40.725905 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.725877 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-kube-api-access-dwbns" (OuterVolumeSpecName: "kube-api-access-dwbns") pod "2557e078-6aa3-4cb4-9fb7-e708d461e1c2" (UID: "2557e078-6aa3-4cb4-9fb7-e708d461e1c2"). InnerVolumeSpecName "kube-api-access-dwbns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:20:40.726010 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.725940 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-dshm" (OuterVolumeSpecName: "dshm") pod "2557e078-6aa3-4cb4-9fb7-e708d461e1c2" (UID: "2557e078-6aa3-4cb4-9fb7-e708d461e1c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:40.726010 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.725949 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2557e078-6aa3-4cb4-9fb7-e708d461e1c2" (UID: "2557e078-6aa3-4cb4-9fb7-e708d461e1c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:20:40.824663 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.824615 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbns\" (UniqueName: \"kubernetes.io/projected/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-kube-api-access-dwbns\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:40.824663 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.824659 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:40.824663 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.824669 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:40.824663 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:40.824677 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2557e078-6aa3-4cb4-9fb7-e708d461e1c2-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:41.244575 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.244507 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" podUID="d803ee27-2fb0-48c5-92f1-279c2326284d" containerName="main" containerID="cri-o://dc1d9e550cd8d8b972d9b4e2547fb7cc299003123caf133409059531dd1cacdd" gracePeriod=45 Aug 01 01:20:41.438601 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.438501 2575 generic.go:356] "Generic (PLEG): container finished" podID="d803ee27-2fb0-48c5-92f1-279c2326284d" containerID="dc1d9e550cd8d8b972d9b4e2547fb7cc299003123caf133409059531dd1cacdd" exitCode=0 Aug 01 01:20:41.438601 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.438586 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" event={"ID":"d803ee27-2fb0-48c5-92f1-279c2326284d","Type":"ContainerDied","Data":"dc1d9e550cd8d8b972d9b4e2547fb7cc299003123caf133409059531dd1cacdd"} Aug 01 01:20:41.439950 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.439921 2575 generic.go:356] "Generic (PLEG): container finished" podID="2557e078-6aa3-4cb4-9fb7-e708d461e1c2" containerID="6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3" exitCode=0 Aug 01 01:20:41.440045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.439953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" event={"ID":"2557e078-6aa3-4cb4-9fb7-e708d461e1c2","Type":"ContainerDied","Data":"6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3"} Aug 01 01:20:41.440045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.439992 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" event={"ID":"2557e078-6aa3-4cb4-9fb7-e708d461e1c2","Type":"ContainerDied","Data":"5af323db5921edaaac13e904498bdd8601e24466869cf9f739b9ca8eaa73ebc4"} Aug 01 01:20:41.440045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.440010 2575 scope.go:117] "RemoveContainer" containerID="6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3" Aug 01 01:20:41.440045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.440020 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc" Aug 01 01:20:41.448995 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.448976 2575 scope.go:117] "RemoveContainer" containerID="6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3" Aug 01 01:20:41.449408 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:20:41.449386 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3\": container with ID starting with 6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3 not found: ID does not exist" containerID="6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3" Aug 01 01:20:41.449482 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.449417 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3"} err="failed to get container status \"6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3\": rpc error: code = NotFound desc = could not find container \"6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3\": container with ID starting with 6829a025fe4dc541d7e09eb89ecd4eef287d42bc654e14b358083c367b30cda3 not found: ID does not exist" Aug 01 01:20:41.456043 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.456010 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc"] Aug 01 01:20:41.459633 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.459609 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/sub-test-model-1e2f5b-kserve-7c875db67c-6dxtc"] Aug 01 01:20:41.496641 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.496613 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:20:41.631009 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.630975 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-tmp-dir\") pod \"d803ee27-2fb0-48c5-92f1-279c2326284d\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " Aug 01 01:20:41.631243 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631026 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-865vl\" (UniqueName: \"kubernetes.io/projected/d803ee27-2fb0-48c5-92f1-279c2326284d-kube-api-access-865vl\") pod \"d803ee27-2fb0-48c5-92f1-279c2326284d\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " Aug 01 01:20:41.631243 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631046 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-dshm\") pod \"d803ee27-2fb0-48c5-92f1-279c2326284d\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " Aug 01 01:20:41.631243 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631060 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-home\") pod \"d803ee27-2fb0-48c5-92f1-279c2326284d\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " Aug 01 01:20:41.631243 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631130 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d803ee27-2fb0-48c5-92f1-279c2326284d-tls-certs\") pod \"d803ee27-2fb0-48c5-92f1-279c2326284d\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " Aug 01 01:20:41.631243 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631197 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-model-cache\") pod \"d803ee27-2fb0-48c5-92f1-279c2326284d\" (UID: \"d803ee27-2fb0-48c5-92f1-279c2326284d\") " Aug 01 01:20:41.631537 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631332 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d803ee27-2fb0-48c5-92f1-279c2326284d" (UID: "d803ee27-2fb0-48c5-92f1-279c2326284d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:41.631537 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631431 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-home" (OuterVolumeSpecName: "home") pod "d803ee27-2fb0-48c5-92f1-279c2326284d" (UID: "d803ee27-2fb0-48c5-92f1-279c2326284d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:41.631537 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.631518 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d803ee27-2fb0-48c5-92f1-279c2326284d" (UID: "d803ee27-2fb0-48c5-92f1-279c2326284d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:41.633328 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.633290 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-dshm" (OuterVolumeSpecName: "dshm") pod "d803ee27-2fb0-48c5-92f1-279c2326284d" (UID: "d803ee27-2fb0-48c5-92f1-279c2326284d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:20:41.633436 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.633361 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d803ee27-2fb0-48c5-92f1-279c2326284d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d803ee27-2fb0-48c5-92f1-279c2326284d" (UID: "d803ee27-2fb0-48c5-92f1-279c2326284d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:20:41.633436 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.633391 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d803ee27-2fb0-48c5-92f1-279c2326284d-kube-api-access-865vl" (OuterVolumeSpecName: "kube-api-access-865vl") pod "d803ee27-2fb0-48c5-92f1-279c2326284d" (UID: "d803ee27-2fb0-48c5-92f1-279c2326284d"). InnerVolumeSpecName "kube-api-access-865vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:20:41.732219 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.732104 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d803ee27-2fb0-48c5-92f1-279c2326284d-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:41.732219 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.732139 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:41.732219 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.732147 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:41.732219 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.732182 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-865vl\" (UniqueName: \"kubernetes.io/projected/d803ee27-2fb0-48c5-92f1-279c2326284d-kube-api-access-865vl\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:41.732219 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.732192 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:41.732219 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:41.732201 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d803ee27-2fb0-48c5-92f1-279c2326284d-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:20:42.446476 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:42.446433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" event={"ID":"d803ee27-2fb0-48c5-92f1-279c2326284d","Type":"ContainerDied","Data":"b4437020414b67226c4358a77f48fd7502989a89cc856b225fe7b99ac7ca67aa"} Aug 01 01:20:42.446476 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:42.446468 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt" Aug 01 01:20:42.447045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:42.446478 2575 scope.go:117] "RemoveContainer" containerID="dc1d9e550cd8d8b972d9b4e2547fb7cc299003123caf133409059531dd1cacdd" Aug 01 01:20:42.467529 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:42.467495 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt"] Aug 01 01:20:42.472789 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:42.472752 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/sub-test-model-4224e1-kserve-5fbc4b5cc9-kp9lt"] Aug 01 01:20:42.915609 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:42.915570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:20:42.963867 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:42.963832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:20:43.162987 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:43.162954 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2557e078-6aa3-4cb4-9fb7-e708d461e1c2" path="/var/lib/kubelet/pods/2557e078-6aa3-4cb4-9fb7-e708d461e1c2/volumes" Aug 01 01:20:43.163370 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:43.163355 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d803ee27-2fb0-48c5-92f1-279c2326284d" path="/var/lib/kubelet/pods/d803ee27-2fb0-48c5-92f1-279c2326284d/volumes" Aug 01 01:20:47.857018 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:47.856974 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:47.857454 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:47.857040 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:47.869875 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:47.869848 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:20:48.486870 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:20:48.486842 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:21:07.647579 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.647546 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd"] Aug 01 01:21:07.648037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.647826 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2557e078-6aa3-4cb4-9fb7-e708d461e1c2" containerName="main" Aug 01 01:21:07.648037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.647836 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2557e078-6aa3-4cb4-9fb7-e708d461e1c2" containerName="main" Aug 01 01:21:07.648037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.647850 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d803ee27-2fb0-48c5-92f1-279c2326284d" containerName="main" Aug 01 01:21:07.648037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.647856 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d803ee27-2fb0-48c5-92f1-279c2326284d" containerName="main" Aug 01 01:21:07.648037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.647913 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2557e078-6aa3-4cb4-9fb7-e708d461e1c2" containerName="main" Aug 01 01:21:07.648037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.647923 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d803ee27-2fb0-48c5-92f1-279c2326284d" containerName="main" Aug 01 01:21:07.650413 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.650398 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.652770 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.652748 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"kube-root-ca.crt\"" Aug 01 01:21:07.652896 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.652752 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"openshift-service-ca.crt\"" Aug 01 01:21:07.653794 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.653777 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"rate-test-model-4224e1-kserve-self-signed-certs\"" Aug 01 01:21:07.653869 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.653855 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4224e1\"/\"default-dockercfg-qpp6c\"" Aug 01 01:21:07.658520 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.658499 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd"] Aug 01 01:21:07.735632 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.735600 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:21:07.739538 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.739516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-tmp-dir\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.739608 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.739547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzq5n\" (UniqueName: \"kubernetes.io/projected/ee8e2963-7136-48db-8036-652470ab5206-kube-api-access-jzq5n\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.739608 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.739570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-home\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.739676 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.739621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8e2963-7136-48db-8036-652470ab5206-tls-certs\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.739676 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.739654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-dshm\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.739742 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.739686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-model-cache\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.840641 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.840604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8e2963-7136-48db-8036-652470ab5206-tls-certs\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.840641 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.840648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-dshm\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.840890 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.840724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-model-cache\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.840890 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.840780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-tmp-dir\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.840890 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.840805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzq5n\" (UniqueName: \"kubernetes.io/projected/ee8e2963-7136-48db-8036-652470ab5206-kube-api-access-jzq5n\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.840890 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.840834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-home\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.841136 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.841116 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-model-cache\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.841236 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.841201 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-tmp-dir\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.841296 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.841238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-home\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.842956 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.842935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-dshm\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.843302 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.843284 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8e2963-7136-48db-8036-652470ab5206-tls-certs\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.848712 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.848692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzq5n\" (UniqueName: \"kubernetes.io/projected/ee8e2963-7136-48db-8036-652470ab5206-kube-api-access-jzq5n\") pod \"rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:07.960651 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:07.960613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:08.098503 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:08.098477 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd"] Aug 01 01:21:08.100551 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:21:08.100522 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee8e2963_7136_48db_8036_652470ab5206.slice/crio-47e7bd58bbe0f36dedd62075acf00442f61ee0cfc8db414ca371ddae75fb69c2 WatchSource:0}: Error finding container 47e7bd58bbe0f36dedd62075acf00442f61ee0cfc8db414ca371ddae75fb69c2: Status 404 returned error can't find the container with id 47e7bd58bbe0f36dedd62075acf00442f61ee0cfc8db414ca371ddae75fb69c2 Aug 01 01:21:08.572369 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:08.572324 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" event={"ID":"ee8e2963-7136-48db-8036-652470ab5206","Type":"ContainerStarted","Data":"72a1936edc6d2231bb4744c06ede6add836008af107e76a8a80ab9dcdd43019d"} Aug 01 01:21:08.572369 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:08.572374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" event={"ID":"ee8e2963-7136-48db-8036-652470ab5206","Type":"ContainerStarted","Data":"47e7bd58bbe0f36dedd62075acf00442f61ee0cfc8db414ca371ddae75fb69c2"} Aug 01 01:21:08.590053 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:08.589994 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" podStartSLOduration=1.589976555 podStartE2EDuration="1.589976555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:21:08.587298997 +0000 UTC m=+3117.944090654" watchObservedRunningTime="2026-08-01 01:21:08.589976555 +0000 UTC m=+3117.946768210" Aug 01 01:21:10.073307 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:10.073269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:21:17.961707 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:17.961656 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:17.962108 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:17.961723 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:17.974355 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:17.974329 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:18.628777 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:18.628744 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:21:48.661927 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:21:48.661850 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:22:00.981615 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:00.981578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:22:11.972311 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:11.972276 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:22:17.861656 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:17.861615 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf"] Aug 01 01:22:17.880776 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:17.880728 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:18.731544 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:18.731506 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd"] Aug 01 01:22:18.761105 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:18.761068 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:21.066137 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:21.066094 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:22:32.881044 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:32.880977 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" podUID="e4dda91f-a228-45e0-a376-082b90101630" containerName="main" containerID="cri-o://485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d" gracePeriod=45 Aug 01 01:22:33.119695 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.119667 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:22:33.271664 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.271612 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-dshm\") pod \"e4dda91f-a228-45e0-a376-082b90101630\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " Aug 01 01:22:33.271664 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.271667 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-tmp-dir\") pod \"e4dda91f-a228-45e0-a376-082b90101630\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " Aug 01 01:22:33.272016 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.271703 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dda91f-a228-45e0-a376-082b90101630-tls-certs\") pod \"e4dda91f-a228-45e0-a376-082b90101630\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " Aug 01 01:22:33.272016 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.271718 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2blx\" (UniqueName: \"kubernetes.io/projected/e4dda91f-a228-45e0-a376-082b90101630-kube-api-access-w2blx\") pod \"e4dda91f-a228-45e0-a376-082b90101630\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " Aug 01 01:22:33.272016 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.271736 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-model-cache\") pod \"e4dda91f-a228-45e0-a376-082b90101630\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " Aug 01 01:22:33.272016 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.271755 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-home\") pod \"e4dda91f-a228-45e0-a376-082b90101630\" (UID: \"e4dda91f-a228-45e0-a376-082b90101630\") " Aug 01 01:22:33.272285 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.272069 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4dda91f-a228-45e0-a376-082b90101630" (UID: "e4dda91f-a228-45e0-a376-082b90101630"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:33.272285 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.272075 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4dda91f-a228-45e0-a376-082b90101630" (UID: "e4dda91f-a228-45e0-a376-082b90101630"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:33.272459 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.272435 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-home" (OuterVolumeSpecName: "home") pod "e4dda91f-a228-45e0-a376-082b90101630" (UID: "e4dda91f-a228-45e0-a376-082b90101630"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:33.273899 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.273873 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-dshm" (OuterVolumeSpecName: "dshm") pod "e4dda91f-a228-45e0-a376-082b90101630" (UID: "e4dda91f-a228-45e0-a376-082b90101630"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:33.274006 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.273896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dda91f-a228-45e0-a376-082b90101630-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4dda91f-a228-45e0-a376-082b90101630" (UID: "e4dda91f-a228-45e0-a376-082b90101630"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:22:33.274006 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.273927 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dda91f-a228-45e0-a376-082b90101630-kube-api-access-w2blx" (OuterVolumeSpecName: "kube-api-access-w2blx") pod "e4dda91f-a228-45e0-a376-082b90101630" (UID: "e4dda91f-a228-45e0-a376-082b90101630"). InnerVolumeSpecName "kube-api-access-w2blx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:22:33.373304 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.373268 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:33.373304 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.373297 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:33.373304 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.373307 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dda91f-a228-45e0-a376-082b90101630-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:33.373546 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.373319 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2blx\" (UniqueName: \"kubernetes.io/projected/e4dda91f-a228-45e0-a376-082b90101630-kube-api-access-w2blx\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:33.373546 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.373328 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:33.373546 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.373337 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dda91f-a228-45e0-a376-082b90101630-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:33.749848 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.749806 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" podUID="ee8e2963-7136-48db-8036-652470ab5206" containerName="main" containerID="cri-o://72a1936edc6d2231bb4744c06ede6add836008af107e76a8a80ab9dcdd43019d" gracePeriod=45 Aug 01 01:22:33.976665 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.976627 2575 generic.go:356] "Generic (PLEG): container finished" podID="ee8e2963-7136-48db-8036-652470ab5206" containerID="72a1936edc6d2231bb4744c06ede6add836008af107e76a8a80ab9dcdd43019d" exitCode=0 Aug 01 01:22:33.977114 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.976698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" event={"ID":"ee8e2963-7136-48db-8036-652470ab5206","Type":"ContainerDied","Data":"72a1936edc6d2231bb4744c06ede6add836008af107e76a8a80ab9dcdd43019d"} Aug 01 01:22:33.978034 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.978009 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4dda91f-a228-45e0-a376-082b90101630" containerID="485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d" exitCode=0 Aug 01 01:22:33.978175 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.978054 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" Aug 01 01:22:33.978175 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.978087 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" event={"ID":"e4dda91f-a228-45e0-a376-082b90101630","Type":"ContainerDied","Data":"485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d"} Aug 01 01:22:33.978175 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.978119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf" event={"ID":"e4dda91f-a228-45e0-a376-082b90101630","Type":"ContainerDied","Data":"c2121fdac7fc9ca1328532acebe97b070c111bd937146968602f6180c5123cda"} Aug 01 01:22:33.978175 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.978141 2575 scope.go:117] "RemoveContainer" containerID="485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d" Aug 01 01:22:33.987658 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.987633 2575 scope.go:117] "RemoveContainer" containerID="485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d" Aug 01 01:22:33.987996 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:22:33.987972 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d\": container with ID starting with 485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d not found: ID does not exist" containerID="485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d" Aug 01 01:22:33.988062 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:33.988009 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d"} err="failed to get container status \"485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d\": rpc error: code = NotFound desc = could not find container \"485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d\": container with ID starting with 485b81f85bacec396a3b5ca6f733f84850f08af0465207334f1e6d91d7164e6d not found: ID does not exist" Aug 01 01:22:34.000124 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.000061 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf"] Aug 01 01:22:34.003907 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.003881 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1e2f5b/rate-test-model-1e2f5b-kserve-746bd745cf-w8wxf"] Aug 01 01:22:34.008585 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.008563 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:22:34.180500 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180466 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzq5n\" (UniqueName: \"kubernetes.io/projected/ee8e2963-7136-48db-8036-652470ab5206-kube-api-access-jzq5n\") pod \"ee8e2963-7136-48db-8036-652470ab5206\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " Aug 01 01:22:34.180684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180515 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-home\") pod \"ee8e2963-7136-48db-8036-652470ab5206\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " Aug 01 01:22:34.180684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180535 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-tmp-dir\") pod \"ee8e2963-7136-48db-8036-652470ab5206\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " Aug 01 01:22:34.180684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180605 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8e2963-7136-48db-8036-652470ab5206-tls-certs\") pod \"ee8e2963-7136-48db-8036-652470ab5206\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " Aug 01 01:22:34.180684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-model-cache\") pod \"ee8e2963-7136-48db-8036-652470ab5206\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " Aug 01 01:22:34.180684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-dshm\") pod \"ee8e2963-7136-48db-8036-652470ab5206\" (UID: \"ee8e2963-7136-48db-8036-652470ab5206\") " Aug 01 01:22:34.180931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180795 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-home" (OuterVolumeSpecName: "home") pod "ee8e2963-7136-48db-8036-652470ab5206" (UID: "ee8e2963-7136-48db-8036-652470ab5206"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:34.180931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180882 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee8e2963-7136-48db-8036-652470ab5206" (UID: "ee8e2963-7136-48db-8036-652470ab5206"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:34.180931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.180898 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee8e2963-7136-48db-8036-652470ab5206" (UID: "ee8e2963-7136-48db-8036-652470ab5206"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:34.182684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.182653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-dshm" (OuterVolumeSpecName: "dshm") pod "ee8e2963-7136-48db-8036-652470ab5206" (UID: "ee8e2963-7136-48db-8036-652470ab5206"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:22:34.182792 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.182702 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8e2963-7136-48db-8036-652470ab5206-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee8e2963-7136-48db-8036-652470ab5206" (UID: "ee8e2963-7136-48db-8036-652470ab5206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:22:34.182792 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.182716 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8e2963-7136-48db-8036-652470ab5206-kube-api-access-jzq5n" (OuterVolumeSpecName: "kube-api-access-jzq5n") pod "ee8e2963-7136-48db-8036-652470ab5206" (UID: "ee8e2963-7136-48db-8036-652470ab5206"). InnerVolumeSpecName "kube-api-access-jzq5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:22:34.282150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.282062 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8e2963-7136-48db-8036-652470ab5206-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:34.282150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.282091 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:34.282150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.282100 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:34.282150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.282108 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzq5n\" (UniqueName: \"kubernetes.io/projected/ee8e2963-7136-48db-8036-652470ab5206-kube-api-access-jzq5n\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:34.282150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.282117 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:34.282150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.282126 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8e2963-7136-48db-8036-652470ab5206-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:22:34.984332 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.984302 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" Aug 01 01:22:34.984831 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.984350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd" event={"ID":"ee8e2963-7136-48db-8036-652470ab5206","Type":"ContainerDied","Data":"47e7bd58bbe0f36dedd62075acf00442f61ee0cfc8db414ca371ddae75fb69c2"} Aug 01 01:22:34.984831 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:34.984414 2575 scope.go:117] "RemoveContainer" containerID="72a1936edc6d2231bb4744c06ede6add836008af107e76a8a80ab9dcdd43019d" Aug 01 01:22:35.006105 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:35.006074 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd"] Aug 01 01:22:35.007860 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:35.007839 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4224e1/rate-test-model-4224e1-kserve-5fd75d8cf-cs8qd"] Aug 01 01:22:35.164003 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:35.163966 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4dda91f-a228-45e0-a376-082b90101630" path="/var/lib/kubelet/pods/e4dda91f-a228-45e0-a376-082b90101630/volumes" Aug 01 01:22:35.164386 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:35.164372 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8e2963-7136-48db-8036-652470ab5206" path="/var/lib/kubelet/pods/ee8e2963-7136-48db-8036-652470ab5206/volumes" Aug 01 01:22:37.163789 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:37.163754 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:38.590963 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:38.590919 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:38.951504 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:38.951470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:39.112995 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:39.112958 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:41.848495 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:41.848451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:42.168661 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:42.168623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:42.265335 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:42.265295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:43.718356 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:43.718313 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:43.955738 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:43.955703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:44.181421 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:44.181379 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:44.275891 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:44.275843 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:22:46.471262 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:22:46.471219 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:23:07.180674 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:07.180636 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:07.856188 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:07.856139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:08.162251 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:08.162219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:08.322287 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:08.322249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:10.975387 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:10.975352 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:11.077841 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:11.077811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:11.176045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:11.176013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:12.268273 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.268235 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg"] Aug 01 01:23:12.268759 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.268518 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dda91f-a228-45e0-a376-082b90101630" containerName="main" Aug 01 01:23:12.268759 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.268529 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dda91f-a228-45e0-a376-082b90101630" containerName="main" Aug 01 01:23:12.268759 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.268537 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8e2963-7136-48db-8036-652470ab5206" containerName="main" Aug 01 01:23:12.268759 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.268543 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8e2963-7136-48db-8036-652470ab5206" containerName="main" Aug 01 01:23:12.268759 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.268598 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8e2963-7136-48db-8036-652470ab5206" containerName="main" Aug 01 01:23:12.268759 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.268608 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dda91f-a228-45e0-a376-082b90101630" containerName="main" Aug 01 01:23:12.271986 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.271963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.274430 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.274400 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-576e07-serving-cert\"" Aug 01 01:23:12.274571 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.274458 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gf2hx\"" Aug 01 01:23:12.275488 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.275472 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:23:12.275595 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.275536 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 01 01:23:12.283359 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.283331 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg"] Aug 01 01:23:12.303551 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.303515 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:12.378426 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.378386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ea186b-8973-47a3-ab59-281f47e6f263-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.378426 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.378431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.378659 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.378465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg9js\" (UniqueName: \"kubernetes.io/projected/a0ea186b-8973-47a3-ab59-281f47e6f263-kube-api-access-bg9js\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.479493 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.479453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ea186b-8973-47a3-ab59-281f47e6f263-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.479698 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.479509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.479698 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.479535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg9js\" (UniqueName: \"kubernetes.io/projected/a0ea186b-8973-47a3-ab59-281f47e6f263-kube-api-access-bg9js\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.479698 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:23:12.479674 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-serving-cert: secret "maas-api-e2e-ipp-b-576e07-serving-cert" not found Aug 01 01:23:12.479873 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:23:12.479763 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls podName:a0ea186b-8973-47a3-ab59-281f47e6f263 nodeName:}" failed. No retries permitted until 2026-08-01 01:23:12.979738112 +0000 UTC m=+3242.336529751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls") pod "maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" (UID: "a0ea186b-8973-47a3-ab59-281f47e6f263") : secret "maas-api-e2e-ipp-b-576e07-serving-cert" not found Aug 01 01:23:12.480215 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.480197 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ea186b-8973-47a3-ab59-281f47e6f263-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.500081 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.500045 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg9js\" (UniqueName: \"kubernetes.io/projected/a0ea186b-8973-47a3-ab59-281f47e6f263-kube-api-access-bg9js\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.642721 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.642621 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:12.785468 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.785437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:12.878657 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.878620 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:12.983105 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.983067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:12.985577 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:12.985550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls\") pod \"maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:13.184539 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:13.184497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:13.321168 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:13.321130 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg"] Aug 01 01:23:13.323188 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:23:13.323145 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ea186b_8973_47a3_ab59_281f47e6f263.slice/crio-a312199e635424c393510f55c0427014b0633c02f810fcf20ead6720d0484c24 WatchSource:0}: Error finding container a312199e635424c393510f55c0427014b0633c02f810fcf20ead6720d0484c24: Status 404 returned error can't find the container with id a312199e635424c393510f55c0427014b0633c02f810fcf20ead6720d0484c24 Aug 01 01:23:13.324336 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:13.324320 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 01 01:23:14.169225 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:14.169188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" event={"ID":"a0ea186b-8973-47a3-ab59-281f47e6f263","Type":"ContainerStarted","Data":"a312199e635424c393510f55c0427014b0633c02f810fcf20ead6720d0484c24"} Aug 01 01:23:15.175483 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:15.175446 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" event={"ID":"a0ea186b-8973-47a3-ab59-281f47e6f263","Type":"ContainerStarted","Data":"46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82"} Aug 01 01:23:15.175885 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:15.175579 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:15.193380 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:15.193334 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" podStartSLOduration=1.697593514 podStartE2EDuration="3.193316204s" totalImagesPullingTime="1.49572269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:15.190869353 +0000 UTC m=+3244.547661006" watchObservedRunningTime="2026-08-01 01:23:15.193316204 +0000 UTC m=+3244.550107859" Aug 01 01:23:15.772252 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:15.772217 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:23:21.186443 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:21.186413 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:23:36.097221 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:36.097184 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:48.138946 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.138904 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn"] Aug 01 01:23:48.143223 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.143199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.146253 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.146228 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2db7f1\"/\"kube-root-ca.crt\"" Aug 01 01:23:48.147574 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.147545 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2db7f1\"/\"default-dockercfg-4jcsk\"" Aug 01 01:23:48.149887 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.149861 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2db7f1\"/\"openshift-service-ca.crt\"" Aug 01 01:23:48.150354 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.150253 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2db7f1\"/\"ipp-route-2db7f1-kserve-self-signed-certs\"" Aug 01 01:23:48.157808 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.157783 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn"] Aug 01 01:23:48.162228 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.162195 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:23:48.181284 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.181255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-dshm\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.181425 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.181289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-model-cache\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.181425 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.181311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-home\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.181425 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.181382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-tmp-dir\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.181425 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.181406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07538e6-f99e-4c7f-86ae-3022174428e2-tls-certs\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.181425 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.181423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr8r\" (UniqueName: \"kubernetes.io/projected/d07538e6-f99e-4c7f-86ae-3022174428e2-kube-api-access-lrr8r\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282386 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-dshm\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282386 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-model-cache\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282644 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-home\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282644 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-tmp-dir\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282644 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07538e6-f99e-4c7f-86ae-3022174428e2-tls-certs\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282644 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr8r\" (UniqueName: \"kubernetes.io/projected/d07538e6-f99e-4c7f-86ae-3022174428e2-kube-api-access-lrr8r\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282913 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-model-cache\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.282980 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282939 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-tmp-dir\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.283032 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.282976 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-home\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.284767 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.284735 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-dshm\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.285037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.285020 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07538e6-f99e-4c7f-86ae-3022174428e2-tls-certs\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.290334 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.290314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr8r\" (UniqueName: \"kubernetes.io/projected/d07538e6-f99e-4c7f-86ae-3022174428e2-kube-api-access-lrr8r\") pod \"ipp-route-2db7f1-kserve-74cf4c664c-xqkwn\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.460922 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.460882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:48.592420 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:48.592395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn"] Aug 01 01:23:48.594197 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:23:48.594166 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd07538e6_f99e_4c7f_86ae_3022174428e2.slice/crio-ccdbb401bf5fcb83a5a91b621182eb500aceb2fd5e27ddf850ae287b61fe541c WatchSource:0}: Error finding container ccdbb401bf5fcb83a5a91b621182eb500aceb2fd5e27ddf850ae287b61fe541c: Status 404 returned error can't find the container with id ccdbb401bf5fcb83a5a91b621182eb500aceb2fd5e27ddf850ae287b61fe541c Aug 01 01:23:49.333592 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:49.333555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" event={"ID":"d07538e6-f99e-4c7f-86ae-3022174428e2","Type":"ContainerStarted","Data":"dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2"} Aug 01 01:23:49.333986 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:49.333604 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" event={"ID":"d07538e6-f99e-4c7f-86ae-3022174428e2","Type":"ContainerStarted","Data":"ccdbb401bf5fcb83a5a91b621182eb500aceb2fd5e27ddf850ae287b61fe541c"} Aug 01 01:23:49.350174 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:49.350107 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" podStartSLOduration=1.3500914019999999 podStartE2EDuration="1.350091402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:23:49.349057941 +0000 UTC m=+3278.705849597" watchObservedRunningTime="2026-08-01 01:23:49.350091402 +0000 UTC m=+3278.706883058" Aug 01 01:23:50.874762 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:50.874727 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:23:58.461245 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:58.461191 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:58.461245 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:58.461258 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:58.473619 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:58.473586 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:23:59.389717 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:23:59.389680 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:24:11.360223 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:11.360196 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:24:11.364004 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:11.363983 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rt596_bf6cd5d5-6fee-4e49-bda7-30a29a317638/ovn-acl-logging/0.log" Aug 01 01:24:16.677938 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:16.677901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:24:27.065895 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:27.065854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:28.366876 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:28.366829 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:29.538810 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:29.538770 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:29.639542 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:29.639507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:30.631876 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.631836 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq"] Aug 01 01:24:30.634949 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.634927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.637483 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.637458 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-659af5-serving-cert\"" Aug 01 01:24:30.644858 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.644829 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq"] Aug 01 01:24:30.669410 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.669372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:30.740782 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.740742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.740782 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.740783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/3d001ee7-bf1b-4a5c-b865-272ec95303d4-kube-api-access-2sjjs\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.741026 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.740811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d001ee7-bf1b-4a5c-b865-272ec95303d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.842134 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.842090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.842350 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.842148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/3d001ee7-bf1b-4a5c-b865-272ec95303d4-kube-api-access-2sjjs\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.842350 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.842217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d001ee7-bf1b-4a5c-b865-272ec95303d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.842350 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:24:30.842274 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-serving-cert: secret "maas-api-e2e-ipp-cleanup-659af5-serving-cert" not found Aug 01 01:24:30.842523 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:24:30.842365 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls podName:3d001ee7-bf1b-4a5c-b865-272ec95303d4 nodeName:}" failed. No retries permitted until 2026-08-01 01:24:31.342341598 +0000 UTC m=+3320.699133249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" (UID: "3d001ee7-bf1b-4a5c-b865-272ec95303d4") : secret "maas-api-e2e-ipp-cleanup-659af5-serving-cert" not found Aug 01 01:24:30.843001 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.842981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d001ee7-bf1b-4a5c-b865-272ec95303d4-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.851430 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.851403 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/3d001ee7-bf1b-4a5c-b865-272ec95303d4-kube-api-access-2sjjs\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:30.918013 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:30.917957 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:31.273910 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:31.273813 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:31.347659 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:31.347616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:31.350091 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:31.350069 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:31.375888 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:31.375838 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:31.549101 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:31.549009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:31.685887 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:31.685861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq"] Aug 01 01:24:31.687621 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:24:31.687597 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d001ee7_bf1b_4a5c_b865_272ec95303d4.slice/crio-13990a3ba01f5fab832e857f85391e87c94a0c4d423606254431ed030852b14e WatchSource:0}: Error finding container 13990a3ba01f5fab832e857f85391e87c94a0c4d423606254431ed030852b14e: Status 404 returned error can't find the container with id 13990a3ba01f5fab832e857f85391e87c94a0c4d423606254431ed030852b14e Aug 01 01:24:32.538653 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:32.538522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" event={"ID":"3d001ee7-bf1b-4a5c-b865-272ec95303d4","Type":"ContainerStarted","Data":"4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e"} Aug 01 01:24:32.538653 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:32.538588 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:32.538653 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:32.538612 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" event={"ID":"3d001ee7-bf1b-4a5c-b865-272ec95303d4","Type":"ContainerStarted","Data":"13990a3ba01f5fab832e857f85391e87c94a0c4d423606254431ed030852b14e"} Aug 01 01:24:32.556747 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:32.556699 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" podStartSLOduration=2.048922157 podStartE2EDuration="2.556682635s" totalImagesPullingTime="507.760478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:24:32.554956511 +0000 UTC m=+3321.911748166" watchObservedRunningTime="2026-08-01 01:24:32.556682635 +0000 UTC m=+3321.913474291" Aug 01 01:24:36.077105 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:36.077068 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:24:38.550864 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:38.550835 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:44.333186 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:44.333135 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:45.039591 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.039543 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln"] Aug 01 01:24:45.043090 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.043072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" Aug 01 01:24:45.046877 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.046853 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68mnr\"" Aug 01 01:24:45.051256 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.051233 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln"] Aug 01 01:24:45.058988 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.058957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6e3cb-dbe0-4a77-966d-0305008d201f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln\" (UID: \"2bc6e3cb-dbe0-4a77-966d-0305008d201f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" Aug 01 01:24:45.159827 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.159792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6e3cb-dbe0-4a77-966d-0305008d201f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln\" (UID: \"2bc6e3cb-dbe0-4a77-966d-0305008d201f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" Aug 01 01:24:45.160599 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.160578 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6e3cb-dbe0-4a77-966d-0305008d201f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln\" (UID: \"2bc6e3cb-dbe0-4a77-966d-0305008d201f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" Aug 01 01:24:45.355647 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.355552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" Aug 01 01:24:45.490823 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.490793 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln"] Aug 01 01:24:45.493576 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:24:45.493549 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc6e3cb_dbe0_4a77_966d_0305008d201f.slice/crio-a36fa27b0cf6f0da8b239d7bfc5755e2f71a06c0e9f3e0a955e93c4b2f64fe43 WatchSource:0}: Error finding container a36fa27b0cf6f0da8b239d7bfc5755e2f71a06c0e9f3e0a955e93c4b2f64fe43: Status 404 returned error can't find the container with id a36fa27b0cf6f0da8b239d7bfc5755e2f71a06c0e9f3e0a955e93c4b2f64fe43 Aug 01 01:24:45.601569 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:45.601536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" event={"ID":"2bc6e3cb-dbe0-4a77-966d-0305008d201f","Type":"ContainerStarted","Data":"a36fa27b0cf6f0da8b239d7bfc5755e2f71a06c0e9f3e0a955e93c4b2f64fe43"} Aug 01 01:24:48.617355 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:48.617261 2575 generic.go:356] "Generic (PLEG): container finished" podID="2bc6e3cb-dbe0-4a77-966d-0305008d201f" containerID="4a49a10035e13aa74de69d1a2807a4f1e2b06112a8c699043bd41fd4ea2974de" exitCode=0 Aug 01 01:24:48.617355 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:48.617321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" event={"ID":"2bc6e3cb-dbe0-4a77-966d-0305008d201f","Type":"ContainerDied","Data":"4a49a10035e13aa74de69d1a2807a4f1e2b06112a8c699043bd41fd4ea2974de"} Aug 01 01:24:49.755941 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:49.755917 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" Aug 01 01:24:49.793806 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:49.793776 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6e3cb-dbe0-4a77-966d-0305008d201f-maas-api-service-ca\") pod \"2bc6e3cb-dbe0-4a77-966d-0305008d201f\" (UID: \"2bc6e3cb-dbe0-4a77-966d-0305008d201f\") " Aug 01 01:24:49.794194 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:49.794148 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc6e3cb-dbe0-4a77-966d-0305008d201f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bc6e3cb-dbe0-4a77-966d-0305008d201f" (UID: "2bc6e3cb-dbe0-4a77-966d-0305008d201f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:24:49.894695 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:49.894621 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6e3cb-dbe0-4a77-966d-0305008d201f-maas-api-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:24:50.628300 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:50.628266 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" event={"ID":"2bc6e3cb-dbe0-4a77-966d-0305008d201f","Type":"ContainerDied","Data":"a36fa27b0cf6f0da8b239d7bfc5755e2f71a06c0e9f3e0a955e93c4b2f64fe43"} Aug 01 01:24:50.628300 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:50.628284 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln" Aug 01 01:24:50.628512 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:50.628315 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a36fa27b0cf6f0da8b239d7bfc5755e2f71a06c0e9f3e0a955e93c4b2f64fe43" Aug 01 01:24:52.541340 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.541301 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq"] Aug 01 01:24:52.541868 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.541839 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" podUID="3d001ee7-bf1b-4a5c-b865-272ec95303d4" containerName="maas-api" containerID="cri-o://4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e" gracePeriod=30 Aug 01 01:24:52.569927 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.569891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:52.634250 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.634203 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:52.693750 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.693708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:52.734684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.734654 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln"] Aug 01 01:24:52.736393 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.736372 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-659af5-lr6ln"] Aug 01 01:24:52.756014 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.755980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:52.795185 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.795107 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:52.818451 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.818417 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d001ee7-bf1b-4a5c-b865-272ec95303d4-openshift-service-ca\") pod \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " Aug 01 01:24:52.818633 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.818606 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls\") pod \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " Aug 01 01:24:52.818708 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.818639 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/3d001ee7-bf1b-4a5c-b865-272ec95303d4-kube-api-access-2sjjs\") pod \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\" (UID: \"3d001ee7-bf1b-4a5c-b865-272ec95303d4\") " Aug 01 01:24:52.819002 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.818940 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d001ee7-bf1b-4a5c-b865-272ec95303d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d001ee7-bf1b-4a5c-b865-272ec95303d4" (UID: "3d001ee7-bf1b-4a5c-b865-272ec95303d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:24:52.821430 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.821390 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d001ee7-bf1b-4a5c-b865-272ec95303d4-kube-api-access-2sjjs" (OuterVolumeSpecName: "kube-api-access-2sjjs") pod "3d001ee7-bf1b-4a5c-b865-272ec95303d4" (UID: "3d001ee7-bf1b-4a5c-b865-272ec95303d4"). InnerVolumeSpecName "kube-api-access-2sjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:24:52.821549 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.821512 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d001ee7-bf1b-4a5c-b865-272ec95303d4" (UID: "3d001ee7-bf1b-4a5c-b865-272ec95303d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:24:52.919853 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.919812 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d001ee7-bf1b-4a5c-b865-272ec95303d4-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:24:52.919853 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.919851 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjjs\" (UniqueName: \"kubernetes.io/projected/3d001ee7-bf1b-4a5c-b865-272ec95303d4-kube-api-access-2sjjs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:24:52.920051 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:52.919866 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d001ee7-bf1b-4a5c-b865-272ec95303d4-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:24:53.102787 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.102694 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:53.163765 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.163724 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc6e3cb-dbe0-4a77-966d-0305008d201f" path="/var/lib/kubelet/pods/2bc6e3cb-dbe0-4a77-966d-0305008d201f/volumes" Aug 01 01:24:53.648334 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.648301 2575 generic.go:356] "Generic (PLEG): container finished" podID="3d001ee7-bf1b-4a5c-b865-272ec95303d4" containerID="4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e" exitCode=0 Aug 01 01:24:53.648813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.648342 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" Aug 01 01:24:53.648813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.648388 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" event={"ID":"3d001ee7-bf1b-4a5c-b865-272ec95303d4","Type":"ContainerDied","Data":"4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e"} Aug 01 01:24:53.648813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.648447 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq" event={"ID":"3d001ee7-bf1b-4a5c-b865-272ec95303d4","Type":"ContainerDied","Data":"13990a3ba01f5fab832e857f85391e87c94a0c4d423606254431ed030852b14e"} Aug 01 01:24:53.648813 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.648482 2575 scope.go:117] "RemoveContainer" containerID="4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e" Aug 01 01:24:53.657521 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.657501 2575 scope.go:117] "RemoveContainer" containerID="4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e" Aug 01 01:24:53.657783 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:24:53.657764 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e\": container with ID starting with 4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e not found: ID does not exist" containerID="4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e" Aug 01 01:24:53.657837 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.657792 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e"} err="failed to get container status \"4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e\": rpc error: code = NotFound desc = could not find container \"4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e\": container with ID starting with 4bf6346e51db940f2155c3fa17e085622435a50dfabd9521494686bfcbe9aa1e not found: ID does not exist" Aug 01 01:24:53.668622 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.668590 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq"] Aug 01 01:24:53.671977 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:53.671942 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-659af5-76bfc447bd-9pbkq"] Aug 01 01:24:55.162482 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:55.162448 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d001ee7-bf1b-4a5c-b865-272ec95303d4" path="/var/lib/kubelet/pods/3d001ee7-bf1b-4a5c-b865-272ec95303d4/volumes" Aug 01 01:24:55.666556 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:55.666520 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:24:58.781494 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:58.781455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:58.882099 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:58.882072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:59.064720 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:59.064631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:24:59.291825 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:24:59.291791 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:02.791546 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.791511 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j"] Aug 01 01:25:02.791931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.791808 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d001ee7-bf1b-4a5c-b865-272ec95303d4" containerName="maas-api" Aug 01 01:25:02.791931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.791818 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d001ee7-bf1b-4a5c-b865-272ec95303d4" containerName="maas-api" Aug 01 01:25:02.791931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.791828 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc6e3cb-dbe0-4a77-966d-0305008d201f" containerName="revoke-keys" Aug 01 01:25:02.791931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.791834 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc6e3cb-dbe0-4a77-966d-0305008d201f" containerName="revoke-keys" Aug 01 01:25:02.791931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.791905 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc6e3cb-dbe0-4a77-966d-0305008d201f" containerName="revoke-keys" Aug 01 01:25:02.791931 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.791913 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d001ee7-bf1b-4a5c-b865-272ec95303d4" containerName="maas-api" Aug 01 01:25:02.796070 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.796052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" Aug 01 01:25:02.798684 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.798665 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68mnr\"" Aug 01 01:25:02.802845 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.802817 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j"] Aug 01 01:25:02.911856 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:02.911813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cbf8e-4905-4153-8faa-c00a0229f87c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j\" (UID: \"cf5cbf8e-4905-4153-8faa-c00a0229f87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" Aug 01 01:25:03.013111 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:03.013066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cbf8e-4905-4153-8faa-c00a0229f87c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j\" (UID: \"cf5cbf8e-4905-4153-8faa-c00a0229f87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" Aug 01 01:25:03.013745 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:03.013725 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cbf8e-4905-4153-8faa-c00a0229f87c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j\" (UID: \"cf5cbf8e-4905-4153-8faa-c00a0229f87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" Aug 01 01:25:03.109385 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:03.109306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" Aug 01 01:25:03.244666 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:03.244625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j"] Aug 01 01:25:03.248208 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:25:03.248178 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5cbf8e_4905_4153_8faa_c00a0229f87c.slice/crio-225ae753bfbad8c9ca789edf83f14d460d55a892297f56da43826046e353f08f WatchSource:0}: Error finding container 225ae753bfbad8c9ca789edf83f14d460d55a892297f56da43826046e353f08f: Status 404 returned error can't find the container with id 225ae753bfbad8c9ca789edf83f14d460d55a892297f56da43826046e353f08f Aug 01 01:25:03.698435 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:03.698340 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" event={"ID":"cf5cbf8e-4905-4153-8faa-c00a0229f87c","Type":"ContainerStarted","Data":"7f56420873c7c202c9ae1a7f36aaad1ca3df0a7b5d1a9da549e3750d72321837"} Aug 01 01:25:03.698435 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:03.698423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" event={"ID":"cf5cbf8e-4905-4153-8faa-c00a0229f87c","Type":"ContainerStarted","Data":"225ae753bfbad8c9ca789edf83f14d460d55a892297f56da43826046e353f08f"} Aug 01 01:25:03.716458 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:03.716394 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" podStartSLOduration=1.7163759349999999 podStartE2EDuration="1.716375935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:25:03.713415953 +0000 UTC m=+3353.070207611" watchObservedRunningTime="2026-08-01 01:25:03.716375935 +0000 UTC m=+3353.073167591" Aug 01 01:25:04.704860 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:04.704825 2575 generic.go:356] "Generic (PLEG): container finished" podID="cf5cbf8e-4905-4153-8faa-c00a0229f87c" containerID="7f56420873c7c202c9ae1a7f36aaad1ca3df0a7b5d1a9da549e3750d72321837" exitCode=0 Aug 01 01:25:04.705294 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:04.704872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" event={"ID":"cf5cbf8e-4905-4153-8faa-c00a0229f87c","Type":"ContainerDied","Data":"7f56420873c7c202c9ae1a7f36aaad1ca3df0a7b5d1a9da549e3750d72321837"} Aug 01 01:25:05.839986 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:05.839959 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" Aug 01 01:25:05.940307 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:05.940257 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cbf8e-4905-4153-8faa-c00a0229f87c-maas-api-service-ca\") pod \"cf5cbf8e-4905-4153-8faa-c00a0229f87c\" (UID: \"cf5cbf8e-4905-4153-8faa-c00a0229f87c\") " Aug 01 01:25:05.940627 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:05.940601 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5cbf8e-4905-4153-8faa-c00a0229f87c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf5cbf8e-4905-4153-8faa-c00a0229f87c" (UID: "cf5cbf8e-4905-4153-8faa-c00a0229f87c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:25:06.041245 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:06.041137 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5cbf8e-4905-4153-8faa-c00a0229f87c-maas-api-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:25:06.715649 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:06.715618 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" event={"ID":"cf5cbf8e-4905-4153-8faa-c00a0229f87c","Type":"ContainerDied","Data":"225ae753bfbad8c9ca789edf83f14d460d55a892297f56da43826046e353f08f"} Aug 01 01:25:06.715827 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:06.715654 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j" Aug 01 01:25:06.715827 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:06.715664 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="225ae753bfbad8c9ca789edf83f14d460d55a892297f56da43826046e353f08f" Aug 01 01:25:11.897089 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:11.896987 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:16.479256 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:16.479213 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:25:18.470422 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:18.470386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:18.566018 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:18.565985 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:18.634348 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:18.634311 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j"] Aug 01 01:25:18.636136 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:18.636103 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:18.636398 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:18.636376 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2db7f1-zlq2j"] Aug 01 01:25:18.688066 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:18.688035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:19.019355 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:19.019319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:19.163150 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:19.163116 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5cbf8e-4905-4153-8faa-c00a0229f87c" path="/var/lib/kubelet/pods/cf5cbf8e-4905-4153-8faa-c00a0229f87c/volumes" Aug 01 01:25:24.003783 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:24.003745 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn"] Aug 01 01:25:24.099353 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:24.099319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:25:25.872345 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:25.872311 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:25:39.022940 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.022894 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" podUID="d07538e6-f99e-4c7f-86ae-3022174428e2" containerName="main" containerID="cri-o://dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2" gracePeriod=45 Aug 01 01:25:39.267678 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.267652 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:25:39.316891 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.316778 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-home\") pod \"d07538e6-f99e-4c7f-86ae-3022174428e2\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " Aug 01 01:25:39.316891 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.316842 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrr8r\" (UniqueName: \"kubernetes.io/projected/d07538e6-f99e-4c7f-86ae-3022174428e2-kube-api-access-lrr8r\") pod \"d07538e6-f99e-4c7f-86ae-3022174428e2\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " Aug 01 01:25:39.316891 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.316885 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-dshm\") pod \"d07538e6-f99e-4c7f-86ae-3022174428e2\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " Aug 01 01:25:39.317213 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.316906 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-model-cache\") pod \"d07538e6-f99e-4c7f-86ae-3022174428e2\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " Aug 01 01:25:39.317213 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.316927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-tmp-dir\") pod \"d07538e6-f99e-4c7f-86ae-3022174428e2\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " Aug 01 01:25:39.317213 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.316964 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07538e6-f99e-4c7f-86ae-3022174428e2-tls-certs\") pod \"d07538e6-f99e-4c7f-86ae-3022174428e2\" (UID: \"d07538e6-f99e-4c7f-86ae-3022174428e2\") " Aug 01 01:25:39.317213 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.317094 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-home" (OuterVolumeSpecName: "home") pod "d07538e6-f99e-4c7f-86ae-3022174428e2" (UID: "d07538e6-f99e-4c7f-86ae-3022174428e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:25:39.317213 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.317186 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "d07538e6-f99e-4c7f-86ae-3022174428e2" (UID: "d07538e6-f99e-4c7f-86ae-3022174428e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:25:39.317463 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.317229 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d07538e6-f99e-4c7f-86ae-3022174428e2" (UID: "d07538e6-f99e-4c7f-86ae-3022174428e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:25:39.317463 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.317241 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-home\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:25:39.319373 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.319348 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07538e6-f99e-4c7f-86ae-3022174428e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d07538e6-f99e-4c7f-86ae-3022174428e2" (UID: "d07538e6-f99e-4c7f-86ae-3022174428e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:25:39.319373 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.319354 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-dshm" (OuterVolumeSpecName: "dshm") pod "d07538e6-f99e-4c7f-86ae-3022174428e2" (UID: "d07538e6-f99e-4c7f-86ae-3022174428e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 01 01:25:39.319523 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.319373 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07538e6-f99e-4c7f-86ae-3022174428e2-kube-api-access-lrr8r" (OuterVolumeSpecName: "kube-api-access-lrr8r") pod "d07538e6-f99e-4c7f-86ae-3022174428e2" (UID: "d07538e6-f99e-4c7f-86ae-3022174428e2"). InnerVolumeSpecName "kube-api-access-lrr8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:25:39.417995 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.417960 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07538e6-f99e-4c7f-86ae-3022174428e2-tls-certs\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:25:39.417995 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.417990 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr8r\" (UniqueName: \"kubernetes.io/projected/d07538e6-f99e-4c7f-86ae-3022174428e2-kube-api-access-lrr8r\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:25:39.417995 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.417999 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-dshm\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:25:39.418277 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.418008 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-model-cache\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:25:39.418277 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.418019 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07538e6-f99e-4c7f-86ae-3022174428e2-tmp-dir\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:25:39.875963 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.875928 2575 generic.go:356] "Generic (PLEG): container finished" podID="d07538e6-f99e-4c7f-86ae-3022174428e2" containerID="dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2" exitCode=0 Aug 01 01:25:39.876151 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.875973 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" Aug 01 01:25:39.876151 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.876016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" event={"ID":"d07538e6-f99e-4c7f-86ae-3022174428e2","Type":"ContainerDied","Data":"dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2"} Aug 01 01:25:39.876151 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.876067 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn" event={"ID":"d07538e6-f99e-4c7f-86ae-3022174428e2","Type":"ContainerDied","Data":"ccdbb401bf5fcb83a5a91b621182eb500aceb2fd5e27ddf850ae287b61fe541c"} Aug 01 01:25:39.876151 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.876086 2575 scope.go:117] "RemoveContainer" containerID="dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2" Aug 01 01:25:39.885336 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.885317 2575 scope.go:117] "RemoveContainer" containerID="dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2" Aug 01 01:25:39.885579 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:25:39.885560 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2\": container with ID starting with dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2 not found: ID does not exist" containerID="dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2" Aug 01 01:25:39.885644 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.885586 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2"} err="failed to get container status \"dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2\": rpc error: code = NotFound desc = could not find container \"dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2\": container with ID starting with dbd87e90f7ac40a9c643758a7e88f1532a85e879b4cdc2bc2a13ab35429baee2 not found: ID does not exist" Aug 01 01:25:39.897332 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.897220 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn"] Aug 01 01:25:39.899997 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:39.899976 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2db7f1/ipp-route-2db7f1-kserve-74cf4c664c-xqkwn"] Aug 01 01:25:41.162699 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:25:41.162664 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d07538e6-f99e-4c7f-86ae-3022174428e2" path="/var/lib/kubelet/pods/d07538e6-f99e-4c7f-86ae-3022174428e2/volumes" Aug 01 01:26:00.364880 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:00.364838 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:00.945020 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:00.944986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:06.370023 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.369990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:06.432571 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.432532 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:06.655364 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.655334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:06.873578 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.873529 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:06.992701 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.992613 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8"] Aug 01 01:26:06.992917 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.992905 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d07538e6-f99e-4c7f-86ae-3022174428e2" containerName="main" Aug 01 01:26:06.992960 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.992918 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d07538e6-f99e-4c7f-86ae-3022174428e2" containerName="main" Aug 01 01:26:06.992960 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.992926 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5cbf8e-4905-4153-8faa-c00a0229f87c" containerName="revoke-keys" Aug 01 01:26:06.992960 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.992932 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5cbf8e-4905-4153-8faa-c00a0229f87c" containerName="revoke-keys" Aug 01 01:26:06.993059 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.992999 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5cbf8e-4905-4153-8faa-c00a0229f87c" containerName="revoke-keys" Aug 01 01:26:06.993059 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.993009 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d07538e6-f99e-4c7f-86ae-3022174428e2" containerName="main" Aug 01 01:26:06.995766 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.995749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" Aug 01 01:26:06.998106 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:06.998085 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68mnr\"" Aug 01 01:26:07.004205 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:07.004181 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8"] Aug 01 01:26:07.152713 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:07.152668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c579f-c3ec-4f45-97da-7fca781f0658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8\" (UID: \"587c579f-c3ec-4f45-97da-7fca781f0658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" Aug 01 01:26:07.253391 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:07.253298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c579f-c3ec-4f45-97da-7fca781f0658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8\" (UID: \"587c579f-c3ec-4f45-97da-7fca781f0658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" Aug 01 01:26:07.253939 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:07.253918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c579f-c3ec-4f45-97da-7fca781f0658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8\" (UID: \"587c579f-c3ec-4f45-97da-7fca781f0658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" Aug 01 01:26:07.309064 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:07.309001 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" Aug 01 01:26:07.436951 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:07.436925 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8"] Aug 01 01:26:07.439328 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:26:07.439298 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587c579f_c3ec_4f45_97da_7fca781f0658.slice/crio-c9955153d7b231783e432389fad521a7052995c2eea773455876a0e4eb564ce5 WatchSource:0}: Error finding container c9955153d7b231783e432389fad521a7052995c2eea773455876a0e4eb564ce5: Status 404 returned error can't find the container with id c9955153d7b231783e432389fad521a7052995c2eea773455876a0e4eb564ce5 Aug 01 01:26:08.012439 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:08.012399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" event={"ID":"587c579f-c3ec-4f45-97da-7fca781f0658","Type":"ContainerStarted","Data":"f7850c07a8345d9014bf38e991bdba1b0d50ed29b30b7e9efab191a8e1b011e6"} Aug 01 01:26:08.012728 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:08.012457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" event={"ID":"587c579f-c3ec-4f45-97da-7fca781f0658","Type":"ContainerStarted","Data":"c9955153d7b231783e432389fad521a7052995c2eea773455876a0e4eb564ce5"} Aug 01 01:26:08.028102 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:08.028043 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" podStartSLOduration=2.028027318 podStartE2EDuration="2.028027318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:26:08.025577947 +0000 UTC m=+3417.382369600" watchObservedRunningTime="2026-08-01 01:26:08.028027318 +0000 UTC m=+3417.384819011" Aug 01 01:26:09.018630 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:09.018594 2575 generic.go:356] "Generic (PLEG): container finished" podID="587c579f-c3ec-4f45-97da-7fca781f0658" containerID="f7850c07a8345d9014bf38e991bdba1b0d50ed29b30b7e9efab191a8e1b011e6" exitCode=0 Aug 01 01:26:09.019002 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:09.018678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" event={"ID":"587c579f-c3ec-4f45-97da-7fca781f0658","Type":"ContainerDied","Data":"f7850c07a8345d9014bf38e991bdba1b0d50ed29b30b7e9efab191a8e1b011e6"} Aug 01 01:26:10.140285 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:10.140257 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" Aug 01 01:26:10.279487 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:10.279395 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c579f-c3ec-4f45-97da-7fca781f0658-maas-api-service-ca\") pod \"587c579f-c3ec-4f45-97da-7fca781f0658\" (UID: \"587c579f-c3ec-4f45-97da-7fca781f0658\") " Aug 01 01:26:10.279772 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:10.279748 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587c579f-c3ec-4f45-97da-7fca781f0658-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "587c579f-c3ec-4f45-97da-7fca781f0658" (UID: "587c579f-c3ec-4f45-97da-7fca781f0658"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:10.380010 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:10.379967 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587c579f-c3ec-4f45-97da-7fca781f0658-maas-api-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:26:11.028142 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:11.028114 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" Aug 01 01:26:11.028334 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:11.028176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8" event={"ID":"587c579f-c3ec-4f45-97da-7fca781f0658","Type":"ContainerDied","Data":"c9955153d7b231783e432389fad521a7052995c2eea773455876a0e4eb564ce5"} Aug 01 01:26:11.028334 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:11.028222 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9955153d7b231783e432389fad521a7052995c2eea773455876a0e4eb564ce5" Aug 01 01:26:17.487631 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.487596 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg"] Aug 01 01:26:17.488059 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.487820 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" podUID="a0ea186b-8973-47a3-ab59-281f47e6f263" containerName="maas-api" containerID="cri-o://46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82" gracePeriod=30 Aug 01 01:26:17.518624 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.518570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:17.619250 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.619208 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:17.684126 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.684091 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8"] Aug 01 01:26:17.687558 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.687498 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:17.688603 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.688580 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-576e07-r2mt8"] Aug 01 01:26:17.738223 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.738149 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:26:17.845077 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.845027 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg9js\" (UniqueName: \"kubernetes.io/projected/a0ea186b-8973-47a3-ab59-281f47e6f263-kube-api-access-bg9js\") pod \"a0ea186b-8973-47a3-ab59-281f47e6f263\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " Aug 01 01:26:17.845077 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.845084 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ea186b-8973-47a3-ab59-281f47e6f263-openshift-service-ca\") pod \"a0ea186b-8973-47a3-ab59-281f47e6f263\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " Aug 01 01:26:17.845363 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.845102 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls\") pod \"a0ea186b-8973-47a3-ab59-281f47e6f263\" (UID: \"a0ea186b-8973-47a3-ab59-281f47e6f263\") " Aug 01 01:26:17.845557 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.845531 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ea186b-8973-47a3-ab59-281f47e6f263-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0ea186b-8973-47a3-ab59-281f47e6f263" (UID: "a0ea186b-8973-47a3-ab59-281f47e6f263"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:17.847230 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.847208 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0ea186b-8973-47a3-ab59-281f47e6f263" (UID: "a0ea186b-8973-47a3-ab59-281f47e6f263"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 01 01:26:17.847303 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.847246 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ea186b-8973-47a3-ab59-281f47e6f263-kube-api-access-bg9js" (OuterVolumeSpecName: "kube-api-access-bg9js") pod "a0ea186b-8973-47a3-ab59-281f47e6f263" (UID: "a0ea186b-8973-47a3-ab59-281f47e6f263"). InnerVolumeSpecName "kube-api-access-bg9js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 01 01:26:17.945723 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.945677 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg9js\" (UniqueName: \"kubernetes.io/projected/a0ea186b-8973-47a3-ab59-281f47e6f263-kube-api-access-bg9js\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:26:17.945723 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.945716 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ea186b-8973-47a3-ab59-281f47e6f263-openshift-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:26:17.945723 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:17.945726 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0ea186b-8973-47a3-ab59-281f47e6f263-maas-api-tls\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:26:18.060743 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.060658 2575 generic.go:356] "Generic (PLEG): container finished" podID="a0ea186b-8973-47a3-ab59-281f47e6f263" containerID="46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82" exitCode=0 Aug 01 01:26:18.060743 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.060703 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" Aug 01 01:26:18.060743 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.060716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" event={"ID":"a0ea186b-8973-47a3-ab59-281f47e6f263","Type":"ContainerDied","Data":"46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82"} Aug 01 01:26:18.060970 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.060762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg" event={"ID":"a0ea186b-8973-47a3-ab59-281f47e6f263","Type":"ContainerDied","Data":"a312199e635424c393510f55c0427014b0633c02f810fcf20ead6720d0484c24"} Aug 01 01:26:18.060970 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.060787 2575 scope.go:117] "RemoveContainer" containerID="46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82" Aug 01 01:26:18.069731 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.069713 2575 scope.go:117] "RemoveContainer" containerID="46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82" Aug 01 01:26:18.070010 ip-10-0-140-111 kubenswrapper[2575]: E0801 01:26:18.069979 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82\": container with ID starting with 46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82 not found: ID does not exist" containerID="46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82" Aug 01 01:26:18.070059 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.070020 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82"} err="failed to get container status \"46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82\": rpc error: code = NotFound desc = could not find container \"46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82\": container with ID starting with 46d843b425872104c57e09b692b760803baac443e2979ecddde8d6a8fc88cd82 not found: ID does not exist" Aug 01 01:26:18.081442 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.081407 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:18.082804 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.082785 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg"] Aug 01 01:26:18.085843 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:18.085821 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-576e07-d6884f46d-k7qkg"] Aug 01 01:26:19.162272 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:19.162239 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587c579f-c3ec-4f45-97da-7fca781f0658" path="/var/lib/kubelet/pods/587c579f-c3ec-4f45-97da-7fca781f0658/volumes" Aug 01 01:26:19.162661 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:19.162615 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ea186b-8973-47a3-ab59-281f47e6f263" path="/var/lib/kubelet/pods/a0ea186b-8973-47a3-ab59-281f47e6f263/volumes" Aug 01 01:26:19.865307 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:19.865275 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:26:23.763612 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:23.763569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:23.826019 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:23.825986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:24.025809 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.025715 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:24.263562 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.263527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:24.409015 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.408985 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6"] Aug 01 01:26:24.409333 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.409318 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ea186b-8973-47a3-ab59-281f47e6f263" containerName="maas-api" Aug 01 01:26:24.409381 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.409336 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ea186b-8973-47a3-ab59-281f47e6f263" containerName="maas-api" Aug 01 01:26:24.409381 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.409348 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587c579f-c3ec-4f45-97da-7fca781f0658" containerName="revoke-keys" Aug 01 01:26:24.409381 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.409353 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="587c579f-c3ec-4f45-97da-7fca781f0658" containerName="revoke-keys" Aug 01 01:26:24.409482 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.409405 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="587c579f-c3ec-4f45-97da-7fca781f0658" containerName="revoke-keys" Aug 01 01:26:24.409482 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.409412 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ea186b-8973-47a3-ab59-281f47e6f263" containerName="maas-api" Aug 01 01:26:24.412113 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.412096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" Aug 01 01:26:24.414654 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.414634 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 01 01:26:24.414756 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.414663 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68mnr\"" Aug 01 01:26:24.419173 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.419133 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6"] Aug 01 01:26:24.603260 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.603210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7a1387-b53b-48e6-bc70-c67278dbf070-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6\" (UID: \"0a7a1387-b53b-48e6-bc70-c67278dbf070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" Aug 01 01:26:24.703940 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.703844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7a1387-b53b-48e6-bc70-c67278dbf070-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6\" (UID: \"0a7a1387-b53b-48e6-bc70-c67278dbf070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" Aug 01 01:26:24.704516 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.704496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7a1387-b53b-48e6-bc70-c67278dbf070-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6\" (UID: \"0a7a1387-b53b-48e6-bc70-c67278dbf070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" Aug 01 01:26:24.723923 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.723891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" Aug 01 01:26:24.862397 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:24.862369 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6"] Aug 01 01:26:24.864373 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:26:24.864340 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7a1387_b53b_48e6_bc70_c67278dbf070.slice/crio-e957f3017e438e798de8ef889173f1b4a9df0f43f06b4d01d995a2f369313438 WatchSource:0}: Error finding container e957f3017e438e798de8ef889173f1b4a9df0f43f06b4d01d995a2f369313438: Status 404 returned error can't find the container with id e957f3017e438e798de8ef889173f1b4a9df0f43f06b4d01d995a2f369313438 Aug 01 01:26:25.096661 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:25.096631 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" event={"ID":"0a7a1387-b53b-48e6-bc70-c67278dbf070","Type":"ContainerStarted","Data":"4e57be7c7cd1f10895505d7fc2f21be7e5118b582859cde17a86996713992778"} Aug 01 01:26:25.096869 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:25.096681 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" event={"ID":"0a7a1387-b53b-48e6-bc70-c67278dbf070","Type":"ContainerStarted","Data":"e957f3017e438e798de8ef889173f1b4a9df0f43f06b4d01d995a2f369313438"} Aug 01 01:26:25.111174 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:25.111110 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" podStartSLOduration=1.1110957830000001 podStartE2EDuration="1.111095783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 01:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:26:25.110055877 +0000 UTC m=+3434.466847533" watchObservedRunningTime="2026-08-01 01:26:25.111095783 +0000 UTC m=+3434.467887438" Aug 01 01:26:26.103305 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:26.103266 2575 generic.go:356] "Generic (PLEG): container finished" podID="0a7a1387-b53b-48e6-bc70-c67278dbf070" containerID="4e57be7c7cd1f10895505d7fc2f21be7e5118b582859cde17a86996713992778" exitCode=0 Aug 01 01:26:26.103691 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:26.103341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" event={"ID":"0a7a1387-b53b-48e6-bc70-c67278dbf070","Type":"ContainerDied","Data":"4e57be7c7cd1f10895505d7fc2f21be7e5118b582859cde17a86996713992778"} Aug 01 01:26:27.238983 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:27.238954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" Aug 01 01:26:27.324035 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:27.323993 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7a1387-b53b-48e6-bc70-c67278dbf070-maas-api-service-ca\") pod \"0a7a1387-b53b-48e6-bc70-c67278dbf070\" (UID: \"0a7a1387-b53b-48e6-bc70-c67278dbf070\") " Aug 01 01:26:27.324373 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:27.324350 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7a1387-b53b-48e6-bc70-c67278dbf070-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a7a1387-b53b-48e6-bc70-c67278dbf070" (UID: "0a7a1387-b53b-48e6-bc70-c67278dbf070"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 01 01:26:27.425320 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:27.425291 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7a1387-b53b-48e6-bc70-c67278dbf070-maas-api-service-ca\") on node \"ip-10-0-140-111.ec2.internal\" DevicePath \"\"" Aug 01 01:26:28.114707 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:28.114668 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" event={"ID":"0a7a1387-b53b-48e6-bc70-c67278dbf070","Type":"ContainerDied","Data":"e957f3017e438e798de8ef889173f1b4a9df0f43f06b4d01d995a2f369313438"} Aug 01 01:26:28.114707 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:28.114693 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6" Aug 01 01:26:28.114933 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:28.114762 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e957f3017e438e798de8ef889173f1b4a9df0f43f06b4d01d995a2f369313438" Aug 01 01:26:34.841045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:34.841011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:34.907651 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:34.907616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:34.962641 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:34.962606 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:35.011294 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:35.011259 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6"] Aug 01 01:26:35.012956 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:35.012935 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1e2f5b-4ffl6"] Aug 01 01:26:35.020801 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:35.020770 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:35.167289 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:35.163053 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7a1387-b53b-48e6-bc70-c67278dbf070" path="/var/lib/kubelet/pods/0a7a1387-b53b-48e6-bc70-c67278dbf070/volumes" Aug 01 01:26:35.382324 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:35.382286 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:37.174908 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:37.174871 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:26:41.350531 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:41.350493 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:41.413782 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:41.413738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:41.612097 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:41.612000 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:41.853129 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:41.853075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:52.532088 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:52.532049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:52.634919 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:52.634876 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:52.717885 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:52.717836 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:52.768855 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:52.768823 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:53.099728 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:53.099697 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:54.661441 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:54.661403 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nh9jc"] Aug 01 01:26:59.171887 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:59.171851 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:59.296383 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:59.296346 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:59.435467 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:59.435357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:26:59.674747 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:26:59.674708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:27:00.137124 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:00.137090 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77fd747845-ddj76_4134e1dd-a10f-4323-a2e9-0fab9f6779d9/authorino/0.log" Aug 01 01:27:06.365785 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:06.365754 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-mtpkd_976f6905-bc5f-4738-8453-ff6bb26b2c32/copy-manifests/0.log" Aug 01 01:27:06.382966 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:06.382941 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-mtpkd_976f6905-bc5f-4738-8453-ff6bb26b2c32/manager/0.log" Aug 01 01:27:07.983520 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:07.983486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77fd747845-ddj76_4134e1dd-a10f-4323-a2e9-0fab9f6779d9/authorino/0.log" Aug 01 01:27:08.191921 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:08.191886 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:27:08.289647 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:08.289546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:27:08.320589 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:08.320562 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-8dplq_0021dd34-6670-459b-8210-889f1546e340/kuadrant-console-plugin/0.log" Aug 01 01:27:08.558045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:08.557955 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-nh9jc_c93cba19-7d5d-42a6-8f31-409aa20e1c54/limitador/0.log" Aug 01 01:27:09.236806 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:09.236764 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6db59d85d8-rtr99_db9fd642-bcbd-487e-95ef-5e3aba0cf39b/kube-auth-proxy/0.log" Aug 01 01:27:10.542353 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:10.542314 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fl9kft_bbe12152-f594-4f89-85a0-26f490f11c97/main/0.log" Aug 01 01:27:10.770509 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:10.770472 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-q85gf_e9666785-eedd-4184-8cc0-c8f663665f8d/main/0.log" Aug 01 01:27:16.040436 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:16.040394 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:27:16.143461 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:16.143423 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" limit=1024 Aug 01 01:27:18.749251 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:18.749221 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gkwsd_6e26feb7-e8a1-43f0-a20d-a18db3aa016b/global-pull-secret-syncer/0.log" Aug 01 01:27:18.890444 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:18.890411 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l7qqq_5e315e32-14a6-4ca3-8015-3593e3d3026c/konnectivity-agent/0.log" Aug 01 01:27:19.028613 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:19.028521 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-111.ec2.internal_7c636173225f22081c0d5c73d018d010/haproxy/0.log" Aug 01 01:27:24.024619 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:24.024576 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77fd747845-ddj76_4134e1dd-a10f-4323-a2e9-0fab9f6779d9/authorino/0.log" Aug 01 01:27:24.127508 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:24.127479 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-8dplq_0021dd34-6670-459b-8210-889f1546e340/kuadrant-console-plugin/0.log" Aug 01 01:27:24.225832 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:24.225808 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-nh9jc_c93cba19-7d5d-42a6-8f31-409aa20e1c54/limitador/0.log" Aug 01 01:27:26.328431 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:26.328401 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lsv64_9932561e-11db-4dc2-856f-35bf72d17c5f/node-exporter/0.log" Aug 01 01:27:26.349947 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:26.349921 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lsv64_9932561e-11db-4dc2-856f-35bf72d17c5f/kube-rbac-proxy/0.log" Aug 01 01:27:26.368891 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:26.368864 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lsv64_9932561e-11db-4dc2-856f-35bf72d17c5f/init-textfile/0.log" Aug 01 01:27:26.806706 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:26.806672 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-769449954f-jdzdt_92ca8f50-32cc-4843-af2d-e6dae49639f3/telemeter-client/0.log" Aug 01 01:27:26.824810 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:26.824782 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-769449954f-jdzdt_92ca8f50-32cc-4843-af2d-e6dae49639f3/reload/0.log" Aug 01 01:27:26.844754 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:26.844725 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-769449954f-jdzdt_92ca8f50-32cc-4843-af2d-e6dae49639f3/kube-rbac-proxy/0.log" Aug 01 01:27:29.114840 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.114796 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb"] Aug 01 01:27:29.115238 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.115124 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7a1387-b53b-48e6-bc70-c67278dbf070" containerName="revoke-keys" Aug 01 01:27:29.115238 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.115136 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7a1387-b53b-48e6-bc70-c67278dbf070" containerName="revoke-keys" Aug 01 01:27:29.115238 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.115198 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7a1387-b53b-48e6-bc70-c67278dbf070" containerName="revoke-keys" Aug 01 01:27:29.117980 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.117963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.120557 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.120537 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bl84r\"/\"default-dockercfg-hwcw7\"" Aug 01 01:27:29.120729 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.120710 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bl84r\"/\"openshift-service-ca.crt\"" Aug 01 01:27:29.121405 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.121391 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bl84r\"/\"kube-root-ca.crt\"" Aug 01 01:27:29.128167 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.128130 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb"] Aug 01 01:27:29.211746 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.211709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-lib-modules\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.211746 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.211757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-proc\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.212006 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.211831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-podres\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.212006 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.211859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr5cf\" (UniqueName: \"kubernetes.io/projected/4063e312-9fed-4594-8974-85054ffeac02-kube-api-access-xr5cf\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.212006 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.211901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-sys\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.312744 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-sys\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.312902 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-lib-modules\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.312902 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-proc\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.312902 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-sys\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.312902 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-podres\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.312902 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr5cf\" (UniqueName: \"kubernetes.io/projected/4063e312-9fed-4594-8974-85054ffeac02-kube-api-access-xr5cf\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.313073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-podres\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.313073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-lib-modules\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.313073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.312979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4063e312-9fed-4594-8974-85054ffeac02-proc\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.321790 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.321761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr5cf\" (UniqueName: \"kubernetes.io/projected/4063e312-9fed-4594-8974-85054ffeac02-kube-api-access-xr5cf\") pod \"perf-node-gather-daemonset-24wnb\" (UID: \"4063e312-9fed-4594-8974-85054ffeac02\") " pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.427424 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.427389 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:29.567814 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:29.567788 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb"] Aug 01 01:27:29.570634 ip-10-0-140-111 kubenswrapper[2575]: W0801 01:27:29.570608 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4063e312_9fed_4594_8974_85054ffeac02.slice/crio-78505c13132c30f8175e7435bd289e9c1f72b1e71e4465146406a4af0197e882 WatchSource:0}: Error finding container 78505c13132c30f8175e7435bd289e9c1f72b1e71e4465146406a4af0197e882: Status 404 returned error can't find the container with id 78505c13132c30f8175e7435bd289e9c1f72b1e71e4465146406a4af0197e882 Aug 01 01:27:30.415393 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:30.415359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" event={"ID":"4063e312-9fed-4594-8974-85054ffeac02","Type":"ContainerStarted","Data":"cb4037b238eb649f01e59eae540b1151d23c97e8d942050a48991c47533b558e"} Aug 01 01:27:30.415818 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:30.415413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" event={"ID":"4063e312-9fed-4594-8974-85054ffeac02","Type":"ContainerStarted","Data":"78505c13132c30f8175e7435bd289e9c1f72b1e71e4465146406a4af0197e882"} Aug 01 01:27:30.415818 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:30.415435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:30.430670 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:30.430622 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" podStartSLOduration=1.430606521 podStartE2EDuration="1.430606521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 01:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 01:27:30.42974072 +0000 UTC m=+3499.786532373" watchObservedRunningTime="2026-08-01 01:27:30.430606521 +0000 UTC m=+3499.787398177" Aug 01 01:27:31.200416 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:31.200388 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rbqbf_afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a/dns/0.log" Aug 01 01:27:31.219234 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:31.219205 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rbqbf_afa9e0f8-6c96-477e-b7ba-f0d1c0ba868a/kube-rbac-proxy/0.log" Aug 01 01:27:31.290240 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:31.290211 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xpfmg_7aa2e9a2-970b-41e5-a04a-9b5b75b5b499/dns-node-resolver/0.log" Aug 01 01:27:31.874757 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:31.874725 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tv49p_131e1d80-8e9d-4665-82f5-b0e3eefeca8c/node-ca/0.log" Aug 01 01:27:32.958175 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:32.958071 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6db59d85d8-rtr99_db9fd642-bcbd-487e-95ef-5e3aba0cf39b/kube-auth-proxy/0.log" Aug 01 01:27:33.909725 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:33.909686 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-48ftq_e1cdfc70-dd52-486f-a9df-2edf6cd2b626/serve-healthcheck-canary/0.log" Aug 01 01:27:34.556035 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:34.556003 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v67t2_e119374b-fa39-4b71-8856-d50e943d4c4c/kube-rbac-proxy/0.log" Aug 01 01:27:34.576045 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:34.576017 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v67t2_e119374b-fa39-4b71-8856-d50e943d4c4c/exporter/0.log" Aug 01 01:27:34.595987 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:34.595954 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v67t2_e119374b-fa39-4b71-8856-d50e943d4c4c/extractor/0.log" Aug 01 01:27:36.429990 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:36.429963 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bl84r/perf-node-gather-daemonset-24wnb" Aug 01 01:27:37.366754 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:37.366727 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-mtpkd_976f6905-bc5f-4738-8453-ff6bb26b2c32/manager/0.log" Aug 01 01:27:37.386762 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:37.386738 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c56689d88-mtpkd_976f6905-bc5f-4738-8453-ff6bb26b2c32/copy-manifests/0.log" Aug 01 01:27:38.779174 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:38.779128 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dd57cb99d-9g67m_c0a414bd-65cf-48c7-992f-a0a21e6620aa/manager/0.log" Aug 01 01:27:39.550772 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:39.550741 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6db59d85d8-rtr99_db9fd642-bcbd-487e-95ef-5e3aba0cf39b/kube-auth-proxy/0.log" Aug 01 01:27:41.231289 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:41.231251 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6db59d85d8-rtr99_db9fd642-bcbd-487e-95ef-5e3aba0cf39b/kube-auth-proxy/0.log" Aug 01 01:27:43.707073 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:43.707037 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-wrj2k_662095f4-3846-4be3-b1c6-3b29fd533d83/migrator/0.log" Aug 01 01:27:43.730299 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:43.730263 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-wrj2k_662095f4-3846-4be3-b1c6-3b29fd533d83/graceful-termination/0.log" Aug 01 01:27:45.368144 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.368112 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7zfn_171e54b7-3ce9-4b0d-96cc-477bdac0a271/kube-multus-additional-cni-plugins/0.log" Aug 01 01:27:45.398037 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.398009 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7zfn_171e54b7-3ce9-4b0d-96cc-477bdac0a271/egress-router-binary-copy/0.log" Aug 01 01:27:45.415707 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.415673 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7zfn_171e54b7-3ce9-4b0d-96cc-477bdac0a271/cni-plugins/0.log" Aug 01 01:27:45.433822 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.433794 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7zfn_171e54b7-3ce9-4b0d-96cc-477bdac0a271/bond-cni-plugin/0.log" Aug 01 01:27:45.453788 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.453760 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7zfn_171e54b7-3ce9-4b0d-96cc-477bdac0a271/routeoverride-cni/0.log" Aug 01 01:27:45.472089 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.472054 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7zfn_171e54b7-3ce9-4b0d-96cc-477bdac0a271/whereabouts-cni-bincopy/0.log" Aug 01 01:27:45.491258 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.491233 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b7zfn_171e54b7-3ce9-4b0d-96cc-477bdac0a271/whereabouts-cni/0.log" Aug 01 01:27:45.741932 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.741899 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zn797_c03487da-17ad-4def-9420-a85d74d06f0b/kube-multus/0.log" Aug 01 01:27:45.800655 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.800624 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kkp8g_84fd6a55-cd43-4cfd-8750-41c249783dd8/network-metrics-daemon/0.log" Aug 01 01:27:45.819197 ip-10-0-140-111 kubenswrapper[2575]: I0801 01:27:45.819149 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kkp8g_84fd6a55-cd43-4cfd-8750-41c249783dd8/kube-rbac-proxy/0.log"