Sep 10 20:54:52.246465 ip-10-0-132-255 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 20:54:52.246608 ip-10-0-132-255 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 20:54:52.246697 ip-10-0-132-255 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 20:54:52.247013 ip-10-0-132-255 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 20:55:02.434561 ip-10-0-132-255 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 20:55:02.434583 ip-10-0-132-255 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 334ab7d37f654e5791b0b958c8664cf2 -- Sep 10 20:56:54.068093 ip-10-0-132-255 systemd[1]: Starting Kubernetes Kubelet... Sep 10 20:56:54.456333 ip-10-0-132-255 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 20:56:54.456333 ip-10-0-132-255 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 20:56:54.456333 ip-10-0-132-255 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 20:56:54.456333 ip-10-0-132-255 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 20:56:54.456333 ip-10-0-132-255 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 20:56:54.457049 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.456418 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 20:56:54.460247 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460231 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:56:54.460247 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460247 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460251 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460254 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460256 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460259 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460262 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460265 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460267 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460270 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460273 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460275 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460278 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460281 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460284 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460287 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460289 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460292 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460294 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460297 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460299 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:56:54.460309 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460302 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460304 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460307 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460309 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460312 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460315 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460318 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460324 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460327 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460331 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460334 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460338 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460343 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460345 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460348 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460351 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460355 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460358 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460361 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:56:54.460845 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460364 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460366 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460369 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460372 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460375 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460377 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460381 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460384 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460387 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460389 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460392 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460394 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460397 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460400 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460403 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460406 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460409 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460411 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460414 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460416 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:56:54.461342 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460419 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460421 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460424 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460428 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460432 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460434 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460437 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460439 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460442 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460444 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460446 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460449 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460451 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460454 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460456 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460459 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460461 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460464 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460467 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:56:54.461854 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460470 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460472 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460475 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460477 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460480 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460482 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460485 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460488 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460490 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460493 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460496 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460498 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460501 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460503 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460506 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460508 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460511 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460513 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460516 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.460518 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:56:54.462306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461221 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461228 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461231 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461234 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461236 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461239 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461242 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461244 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461247 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461249 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461252 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461255 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461257 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461260 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461263 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461268 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461272 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461275 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461278 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461280 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:56:54.462800 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461284 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461287 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461290 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461293 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461296 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461299 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461302 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461304 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461307 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461310 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461312 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461315 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461318 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461320 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461322 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461326 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461329 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461332 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461334 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:56:54.463317 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461337 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461345 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461348 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461351 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461353 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461356 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461359 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461362 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461365 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461367 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461369 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461372 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461374 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461377 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461381 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461385 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461387 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461391 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461393 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461396 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:56:54.463775 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461399 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461401 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461404 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461406 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461409 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461411 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461414 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461416 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461420 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461422 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461425 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461429 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461432 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461435 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461438 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461441 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461444 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461447 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461449 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461452 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:56:54.464298 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461455 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461457 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461459 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461462 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461464 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461467 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461470 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461472 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461475 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461477 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461480 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461482 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461485 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461487 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461490 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461492 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461494 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461497 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461499 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461502 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:56:54.464780 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461862 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461867 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461912 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461916 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461920 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461923 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461926 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461929 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461932 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461935 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461937 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461940 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461943 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461946 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461948 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461951 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461954 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461958 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461962 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:56:54.465276 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461965 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461967 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461970 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461973 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461975 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461978 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461981 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461984 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461987 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461989 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461992 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461994 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461997 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.461999 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462002 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462004 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462007 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462009 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462012 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462014 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:56:54.465765 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462017 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462019 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462022 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462024 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462026 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462029 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462032 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462036 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462040 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462043 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462045 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462048 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462050 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462052 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462055 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462057 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462060 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462062 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462065 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:56:54.466265 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462067 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462070 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462073 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462075 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462078 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462081 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462083 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462085 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462088 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462090 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462093 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462095 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462097 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462100 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462102 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462104 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462107 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462110 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462112 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462115 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:56:54.466718 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462118 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462120 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462123 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462125 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462128 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462131 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462133 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462136 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462138 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462140 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462143 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462145 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462147 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462152 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462154 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462157 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462159 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462162 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462164 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462166 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:56:54.467249 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462170 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462241 2574 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462273 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462281 2574 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462286 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462290 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462293 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462297 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462301 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462304 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462307 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462310 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462314 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462318 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462321 2574 flags.go:64] FLAG: --cgroup-root="" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462324 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462327 2574 flags.go:64] FLAG: --client-ca-file="" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462330 2574 flags.go:64] FLAG: --cloud-provider="external" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462332 2574 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462336 2574 flags.go:64] FLAG: --cluster-domain="" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462339 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462342 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462345 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462348 2574 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 20:56:54.467723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462351 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462356 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462359 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462363 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462365 2574 flags.go:64] FLAG: --contention-profiling="false" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462368 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462371 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462374 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462377 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462381 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462383 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462386 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462389 2574 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462391 2574 flags.go:64] FLAG: --enable-server="true" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462394 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462399 2574 flags.go:64] FLAG: --event-burst="100" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462401 2574 flags.go:64] FLAG: --event-qps="50" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462404 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462407 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462410 2574 flags.go:64] FLAG: --eviction-hard="" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462413 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462416 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462420 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462423 2574 flags.go:64] FLAG: --eviction-soft="" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462426 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 20:56:54.468313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462429 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462432 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462435 2574 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462437 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462440 2574 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462443 2574 flags.go:64] FLAG: --feature-gates="" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462446 2574 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462449 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462455 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462458 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462461 2574 flags.go:64] FLAG: --healthz-port="10248" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462464 2574 flags.go:64] FLAG: --help="false" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462467 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462470 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462472 2574 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462475 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462478 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462481 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462484 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462487 2574 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462490 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462492 2574 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462495 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462498 2574 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 20:56:54.469051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462501 2574 flags.go:64] FLAG: --kube-reserved="" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462503 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462506 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462509 2574 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462512 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462515 2574 flags.go:64] FLAG: --lock-file="" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462520 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462523 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462526 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462532 2574 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462534 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462537 2574 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462540 2574 flags.go:64] FLAG: --logging-format="text" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462543 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462546 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462549 2574 flags.go:64] FLAG: --manifest-url="" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462552 2574 flags.go:64] FLAG: --manifest-url-header="" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462557 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462560 2574 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462564 2574 flags.go:64] FLAG: --max-pods="110" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462567 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462569 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462572 2574 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462575 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462578 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 20:56:54.470098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462581 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462584 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462590 2574 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462593 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462596 2574 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462599 2574 flags.go:64] FLAG: --pod-cidr="" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462601 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462606 2574 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462609 2574 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462612 2574 flags.go:64] FLAG: --pods-per-core="0" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462615 2574 flags.go:64] FLAG: --port="10250" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462618 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462621 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f054e994a67635ff" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462624 2574 flags.go:64] FLAG: --qos-reserved="" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462627 2574 flags.go:64] FLAG: --read-only-port="10255" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462630 2574 flags.go:64] FLAG: --register-node="true" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462633 2574 flags.go:64] FLAG: --register-with-taints="" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462637 2574 flags.go:64] FLAG: --registry-burst="10" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462639 2574 flags.go:64] FLAG: --registry-qps="5" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462642 2574 flags.go:64] FLAG: --reserved-cpus="" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462644 2574 flags.go:64] FLAG: --reserved-memory="" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462648 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462651 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462653 2574 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462658 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 20:56:54.471187 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462661 2574 flags.go:64] FLAG: --runonce="false" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462663 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462668 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462671 2574 flags.go:64] FLAG: --seccomp-default="false" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462674 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462677 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462680 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462682 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462685 2574 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462688 2574 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462691 2574 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462694 2574 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462696 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462699 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462702 2574 flags.go:64] FLAG: --system-cgroups="" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462705 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462710 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462713 2574 flags.go:64] FLAG: --tls-cert-file="" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462715 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462719 2574 flags.go:64] FLAG: --tls-min-version="" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462722 2574 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462724 2574 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462734 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462736 2574 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462739 2574 flags.go:64] FLAG: --v="2" Sep 10 20:56:54.471976 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462743 2574 flags.go:64] FLAG: --version="false" Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462747 2574 flags.go:64] FLAG: --vmodule="" Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462750 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.462753 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462911 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462916 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462920 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462923 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462926 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462930 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462933 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462936 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462938 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462941 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462944 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462947 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462950 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462952 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462955 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462958 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:56:54.473056 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462960 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462963 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462966 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462968 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462971 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462973 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462976 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462978 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462981 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462983 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462986 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462989 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462992 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462994 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462996 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.462999 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463003 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463007 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463012 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:56:54.473655 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463016 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463019 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463024 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463026 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463029 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463032 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463035 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463040 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463044 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463047 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463049 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463052 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463060 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463063 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463066 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463068 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463071 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463073 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463076 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463079 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:56:54.474452 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463081 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463084 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463086 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463089 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463096 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463099 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463101 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463104 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463106 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463109 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463111 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463115 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463117 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463120 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463124 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463127 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463129 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463132 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463134 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463137 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:56:54.475246 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463139 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463142 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463144 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463147 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463149 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463152 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463154 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463158 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463161 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463164 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463166 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463168 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463171 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463174 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463176 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463179 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463182 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463191 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463193 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:56:54.475955 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463196 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463198 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463201 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463203 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.463207 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.463215 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.471031 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.471049 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.471768 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.471782 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471893 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471902 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471907 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471912 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471917 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:56:54.476454 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471921 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471926 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471930 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471934 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471938 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471942 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471946 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471951 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471955 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471959 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471963 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471967 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471972 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471975 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471979 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471984 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471987 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471991 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471995 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.471999 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:56:54.476871 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472003 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472007 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472011 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472015 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472021 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472026 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472030 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472034 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472038 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472042 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472046 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472051 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472055 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472059 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472063 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472067 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472071 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472076 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472080 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472085 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:56:54.477395 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472089 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472093 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472097 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472101 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472105 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472109 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472114 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472118 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472123 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472127 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472130 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472135 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472141 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472147 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472152 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472157 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472163 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472168 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472172 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:56:54.477973 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472177 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472181 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472185 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472190 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472194 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472200 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472205 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472209 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472214 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472218 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472222 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472226 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472230 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472234 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472238 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472243 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472247 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472251 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472255 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:56:54.478460 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472259 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472263 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472270 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472275 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472280 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472284 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472288 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472292 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472297 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472301 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472306 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472310 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472315 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472319 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472323 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:56:54.478974 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472327 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.472335 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472535 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472544 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472549 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472554 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472558 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472562 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472567 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472571 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472576 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472580 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472585 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472589 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472593 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 20:56:54.479485 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472598 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472602 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472606 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472610 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472614 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472618 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472622 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472626 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472630 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472634 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472638 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472642 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472646 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472651 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472657 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472664 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472669 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472673 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472677 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 20:56:54.480132 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472682 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472685 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472691 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472697 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472702 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472706 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472710 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472714 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472718 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472722 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472726 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472730 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472734 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472738 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472742 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472747 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472751 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472755 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472759 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 20:56:54.480904 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472763 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472769 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472774 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472778 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472783 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472787 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472792 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472796 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472800 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472821 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472827 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472830 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472834 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472839 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472843 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472846 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472850 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472854 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472858 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472862 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 20:56:54.481553 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472866 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472870 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472874 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472878 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472882 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472886 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472890 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472894 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472898 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472902 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472906 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472910 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472914 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472918 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472922 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472926 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472930 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472934 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472939 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472943 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 20:56:54.482186 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472947 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472951 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472955 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472960 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472964 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472968 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472972 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:54.472975 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.472983 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.473132 2574 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.475721 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.476845 2574 server.go:1019] "Starting client certificate rotation" Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.476952 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 20:56:54.482673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.477661 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 20:56:54.483021 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.481264 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 20:56:54.484652 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.484631 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 20:56:54.494295 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.494278 2574 log.go:25] "Validated CRI v1 runtime API" Sep 10 20:56:54.497316 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.497302 2574 log.go:25] "Validated CRI v1 image API" Sep 10 20:56:54.498177 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.498158 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 20:56:54.500428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.500409 2574 fs.go:135] Filesystem UUIDs: map[4732353c-83b9-402a-a27d-c02a8e545634:/dev/nvme0n1p4 506929a8-cb51-40d4-9963-c6ba1ee43c18:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 10 20:56:54.500494 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.500426 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 20:56:54.506025 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.505925 2574 manager.go:216] Machine: {Timestamp:2026-09-10 20:56:54.504727239 +0000 UTC m=+0.323301352 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2996753 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b19d2c078a4aaf6efe40a71517b14 SystemUUID:ec2b19d2-c078-a4aa-f6ef-e40a71517b14 BootID:334ab7d3-7f65-4e57-91b0-b958c8664cf2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:c7:5d:26:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:c7:5d:26:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:59:07:d2:32:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 20:56:54.506025 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.506024 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 20:56:54.506141 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.506130 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 20:56:54.508929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.508906 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 20:56:54.509082 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.508931 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-255.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 20:56:54.509621 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.509609 2574 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 20:56:54.509675 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.509628 2574 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 20:56:54.509675 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.509638 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 20:56:54.510170 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.510159 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 20:56:54.510921 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.510910 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 20:56:54.514556 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.514542 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 10 20:56:54.514694 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.514683 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 20:56:54.514760 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.514733 2574 kubelet.go:486] "Attempting to sync node with API server" Sep 10 20:56:54.514760 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.514748 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 20:56:54.514867 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.514768 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 20:56:54.514867 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.514781 2574 kubelet.go:398] "Adding apiserver pod source" Sep 10 20:56:54.514867 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.514793 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 20:56:54.515394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.515380 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 20:56:54.515474 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.515413 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 20:56:54.518927 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.518902 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 20:56:54.519020 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.518957 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 20:56:54.521001 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.520985 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 20:56:54.521060 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.521021 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 20:56:54.522575 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522563 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 20:56:54.522628 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522581 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 20:56:54.522628 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522589 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 20:56:54.522628 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522595 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 20:56:54.522628 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522606 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 20:56:54.522628 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522614 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 20:56:54.522628 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522629 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 20:56:54.522790 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522637 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 20:56:54.522790 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522650 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 20:56:54.522790 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522657 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 20:56:54.522790 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522668 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 20:56:54.522790 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.522675 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 20:56:54.523240 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.523230 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 20:56:54.523271 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.523240 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 20:56:54.527165 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.527148 2574 server.go:1268] "Started kubelet" Sep 10 20:56:54.527293 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.527243 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 20:56:54.527331 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.527250 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 20:56:54.527372 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.527336 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 20:56:54.527975 ip-10-0-132-255 systemd[1]: Started Kubernetes Kubelet. Sep 10 20:56:54.529178 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.529165 2574 server.go:310] "Adding debug handlers to kubelet server" Sep 10 20:56:54.529493 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.529476 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 20:56:54.532823 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.532791 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-255.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 20:56:54.532823 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.532791 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 20:56:54.533597 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.533580 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 20:56:54.534608 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.534591 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 20:56:54.535192 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.535175 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 20:56:54.535286 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.535263 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:54.535712 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.535672 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 20:56:54.535712 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.535676 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 20:56:54.535712 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.535703 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 20:56:54.535926 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.535842 2574 reconstruct.go:97] "Volume reconstruction finished" Sep 10 20:56:54.535926 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.535850 2574 reconciler.go:29] "Reconciler: start to sync state" Sep 10 20:56:54.536066 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536053 2574 factory.go:55] Registering systemd factory Sep 10 20:56:54.536102 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536071 2574 factory.go:223] Registration of the systemd container factory successfully Sep 10 20:56:54.536259 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536249 2574 factory.go:153] Registering CRI-O factory Sep 10 20:56:54.536299 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536260 2574 factory.go:223] Registration of the crio container factory successfully Sep 10 20:56:54.536329 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536305 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 20:56:54.536329 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536317 2574 factory.go:103] Registering Raw factory Sep 10 20:56:54.536400 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536336 2574 manager.go:1196] Started watching for new ooms in manager Sep 10 20:56:54.536779 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.536767 2574 manager.go:349] Starting recovery of all containers Sep 10 20:56:54.548721 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.548699 2574 manager.go:354] Recovery completed Sep 10 20:56:54.549480 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.549447 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 20:56:54.549480 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.549455 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 20:56:54.549693 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.549659 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 20:56:54.550304 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.550286 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 10 20:56:54.550891 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.549781 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-255.ec2.internal.18d41127c7d7256a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-255.ec2.internal,UID:ip-10-0-132-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-255.ec2.internal,},FirstTimestamp:2026-09-10 20:56:54.527092074 +0000 UTC m=+0.345666185,LastTimestamp:2026-09-10 20:56:54.527092074 +0000 UTC m=+0.345666185,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-255.ec2.internal,}" Sep 10 20:56:54.553474 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.553462 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:54.555569 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.555543 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:54.555623 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.555583 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:54.555623 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.555596 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:54.556159 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.556147 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 20:56:54.556159 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.556157 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 20:56:54.556159 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.556159 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:54.556282 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.556172 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 10 20:56:54.557058 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.557043 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:54.557115 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.557068 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:54.557115 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.557080 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:54.558190 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.558131 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-255.ec2.internal.18d41127c989a743 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-255.ec2.internal,UID:ip-10-0-132-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-255.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-255.ec2.internal,},FirstTimestamp:2026-09-10 20:56:54.555567939 +0000 UTC m=+0.374142050,LastTimestamp:2026-09-10 20:56:54.555567939 +0000 UTC m=+0.374142050,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-255.ec2.internal,}" Sep 10 20:56:54.558316 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.558306 2574 policy_none.go:49] "None policy: Start" Sep 10 20:56:54.558356 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.558321 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 20:56:54.558356 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.558333 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 20:56:54.560365 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.560354 2574 policy_none.go:47] "Start" Sep 10 20:56:54.572861 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.572782 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-255.ec2.internal.18d41127c989ff49 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-255.ec2.internal,UID:ip-10-0-132-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-255.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-255.ec2.internal,},FirstTimestamp:2026-09-10 20:56:54.555590473 +0000 UTC m=+0.374164583,LastTimestamp:2026-09-10 20:56:54.555590473 +0000 UTC m=+0.374164583,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-255.ec2.internal,}" Sep 10 20:56:54.590318 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.590148 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-255.ec2.internal.18d41127c98a2381 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-255.ec2.internal,UID:ip-10-0-132-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-255.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-255.ec2.internal,},FirstTimestamp:2026-09-10 20:56:54.555599745 +0000 UTC m=+0.374173857,LastTimestamp:2026-09-10 20:56:54.555599745 +0000 UTC m=+0.374173857,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-255.ec2.internal,}" Sep 10 20:56:54.597484 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.597466 2574 manager.go:337] "Starting Device Plugin manager" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.597514 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.597524 2574 server.go:87] "Starting device plugin registration server" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.597761 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.597771 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.597878 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.597961 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.597969 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.598467 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.598507 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.601653 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-255.ec2.internal.18d41127c989a743\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-255.ec2.internal.18d41127c989a743 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-255.ec2.internal,UID:ip-10-0-132-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-255.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-255.ec2.internal,},FirstTimestamp:2026-09-10 20:56:54.555567939 +0000 UTC m=+0.374142050,LastTimestamp:2026-09-10 20:56:54.55705635 +0000 UTC m=+0.375630462,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-255.ec2.internal,}" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.611643 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-255.ec2.internal.18d41127c989ff49\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-255.ec2.internal.18d41127c989ff49 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-255.ec2.internal,UID:ip-10-0-132-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-255.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-255.ec2.internal,},FirstTimestamp:2026-09-10 20:56:54.555590473 +0000 UTC m=+0.374164583,LastTimestamp:2026-09-10 20:56:54.557073524 +0000 UTC m=+0.375647635,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-255.ec2.internal,}" Sep 10 20:56:54.622929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.614989 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kml2h" Sep 10 20:56:54.626974 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.626958 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kml2h" Sep 10 20:56:54.634957 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.634901 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-255.ec2.internal.18d41127c98a2381\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-255.ec2.internal.18d41127c98a2381 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-255.ec2.internal,UID:ip-10-0-132-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-255.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-255.ec2.internal,},FirstTimestamp:2026-09-10 20:56:54.555599745 +0000 UTC m=+0.374173857,LastTimestamp:2026-09-10 20:56:54.557085813 +0000 UTC m=+0.375659926,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-255.ec2.internal,}" Sep 10 20:56:54.642525 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.642498 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 20:56:54.643710 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.643695 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 20:56:54.643775 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.643742 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 20:56:54.643775 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.643771 2574 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 20:56:54.643879 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.643820 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 20:56:54.652731 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.652717 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:56:54.698236 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.698206 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:54.699441 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.699425 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:54.699507 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.699458 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:54.699507 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.699468 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:54.699507 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.699493 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.709644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.709604 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.709644 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.709625 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-255.ec2.internal\": node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:54.744201 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.744180 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal"] Sep 10 20:56:54.744260 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.744237 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:54.745006 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.744991 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:54.745063 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.745030 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:54.745063 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.745040 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:54.746279 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.746265 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:54.747282 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.747270 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:54.747386 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.747375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.747420 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.747405 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:54.747988 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.747969 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:54.748047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.747991 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:54.748047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.748000 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:54.748047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.748011 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:54.748047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.748015 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:54.748047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.748030 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:54.750082 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.750052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.750173 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.750092 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:54.750687 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.750675 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:54.750742 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.750706 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:54.750742 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.750719 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:54.771377 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.771361 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-255.ec2.internal\" not found" node="ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.774709 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.774693 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-255.ec2.internal\" not found" node="ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.837602 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.837584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f953879e24b63da8b00052cd2968abb-config\") pod \"kube-apiserver-proxy-ip-10-0-132-255.ec2.internal\" (UID: \"0f953879e24b63da8b00052cd2968abb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.837684 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.837606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a062f58674a98f4e1b68c4c7aec1dbff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal\" (UID: \"a062f58674a98f4e1b68c4c7aec1dbff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.837684 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.837624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a062f58674a98f4e1b68c4c7aec1dbff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal\" (UID: \"a062f58674a98f4e1b68c4c7aec1dbff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.846950 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.846931 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:54.938477 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.938447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a062f58674a98f4e1b68c4c7aec1dbff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal\" (UID: \"a062f58674a98f4e1b68c4c7aec1dbff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.938477 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.938479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a062f58674a98f4e1b68c4c7aec1dbff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal\" (UID: \"a062f58674a98f4e1b68c4c7aec1dbff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.938658 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.938503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f953879e24b63da8b00052cd2968abb-config\") pod \"kube-apiserver-proxy-ip-10-0-132-255.ec2.internal\" (UID: \"0f953879e24b63da8b00052cd2968abb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.938658 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.938556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a062f58674a98f4e1b68c4c7aec1dbff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal\" (UID: \"a062f58674a98f4e1b68c4c7aec1dbff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.938658 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.938564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a062f58674a98f4e1b68c4c7aec1dbff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal\" (UID: \"a062f58674a98f4e1b68c4c7aec1dbff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.938658 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:54.938596 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f953879e24b63da8b00052cd2968abb-config\") pod \"kube-apiserver-proxy-ip-10-0-132-255.ec2.internal\" (UID: \"0f953879e24b63da8b00052cd2968abb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" Sep 10 20:56:54.947580 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:54.947556 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.048649 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.048559 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.073766 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.073740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:55.077084 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.077060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" Sep 10 20:56:55.149090 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.149055 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.249568 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.249539 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.350066 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.350003 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.361680 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.361661 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:56:55.450302 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.450273 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.464025 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.464002 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:56:55.477819 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.477786 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 20:56:55.477926 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.477908 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 20:56:55.477977 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.477936 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 20:56:55.477977 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.477933 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 20:56:55.550730 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.550704 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.596842 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:55.596791 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda062f58674a98f4e1b68c4c7aec1dbff.slice/crio-e17ad937fd7a63c469f08d5d793c59435ecef107344ffb0b58b791fdb4b34475 WatchSource:0}: Error finding container e17ad937fd7a63c469f08d5d793c59435ecef107344ffb0b58b791fdb4b34475: Status 404 returned error can't find the container with id e17ad937fd7a63c469f08d5d793c59435ecef107344ffb0b58b791fdb4b34475 Sep 10 20:56:55.597306 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:55.597285 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f953879e24b63da8b00052cd2968abb.slice/crio-8ac5a74c5bf7fd018e13c765eb9f3fa974b7f6e908f77210773ce6a928409df4 WatchSource:0}: Error finding container 8ac5a74c5bf7fd018e13c765eb9f3fa974b7f6e908f77210773ce6a928409df4: Status 404 returned error can't find the container with id 8ac5a74c5bf7fd018e13c765eb9f3fa974b7f6e908f77210773ce6a928409df4 Sep 10 20:56:55.602091 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.602072 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 20:56:55.632071 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.632021 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:51:55 +0000 UTC" deadline="2028-02-25 02:57:20.928659694 +0000 UTC" Sep 10 20:56:55.632071 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.632064 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12774h0m25.296598418s" Sep 10 20:56:55.647128 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.647089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" event={"ID":"a062f58674a98f4e1b68c4c7aec1dbff","Type":"ContainerStarted","Data":"e17ad937fd7a63c469f08d5d793c59435ecef107344ffb0b58b791fdb4b34475"} Sep 10 20:56:55.648206 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.648187 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" event={"ID":"0f953879e24b63da8b00052cd2968abb","Type":"ContainerStarted","Data":"8ac5a74c5bf7fd018e13c765eb9f3fa974b7f6e908f77210773ce6a928409df4"} Sep 10 20:56:55.651331 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.651318 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.672475 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.672455 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 20:56:55.673514 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.673496 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientMemory" Sep 10 20:56:55.673591 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.673526 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 20:56:55.673591 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.673536 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeHasSufficientPID" Sep 10 20:56:55.751627 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.751603 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.852181 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:55.852123 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-255.ec2.internal\" not found" Sep 10 20:56:55.900636 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.900612 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:56:55.936283 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.936259 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" Sep 10 20:56:55.961779 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.961760 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 20:56:55.962767 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.962754 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" Sep 10 20:56:55.975395 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:55.975379 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 20:56:56.516394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.516107 2574 apiserver.go:52] "Watching apiserver" Sep 10 20:56:56.525492 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.525289 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 20:56:56.526836 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.526452 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal","openshift-multus/network-metrics-daemon-kflkv","openshift-network-operator/iptables-alerter-dnbmd","openshift-ovn-kubernetes/ovnkube-node-xkwq7","kube-system/global-pull-secret-syncer-jsfgg","kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal","openshift-dns/node-resolver-tc7ff","openshift-image-registry/node-ca-pmgvp","openshift-multus/multus-additional-cni-plugins-82h58","openshift-multus/multus-t57dn","openshift-network-diagnostics/network-check-target-6rlj6","kube-system/konnectivity-agent-gfcs9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz","openshift-cluster-node-tuning-operator/tuned-2nl69"] Sep 10 20:56:56.529096 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.529070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.532144 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.531801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:56.532144 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.531877 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:56:56.533789 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.533126 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 20:56:56.533789 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.533389 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 20:56:56.533789 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.533626 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfnq\"" Sep 10 20:56:56.534908 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.534553 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.535931 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.535468 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 20:56:56.538183 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.536961 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:56:56.538183 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.537263 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jzsrp\"" Sep 10 20:56:56.538183 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.537516 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 20:56:56.540439 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.538966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.541834 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.541451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.542845 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.542520 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 20:56:56.543322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.543295 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 20:56:56.543430 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.543323 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 20:56:56.543603 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.543586 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 20:56:56.543664 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.543602 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tcwhr\"" Sep 10 20:56:56.548491 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.548468 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 20:56:56.548808 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.548791 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 20:56:56.549110 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.549093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.550449 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.550428 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 20:56:56.551130 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.551111 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bmjc\"" Sep 10 20:56:56.551730 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.551710 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 20:56:56.552009 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.551985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-tmp-dir\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.552091 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-ovnkube-script-lib\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552091 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4wk\" (UniqueName: \"kubernetes.io/projected/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-kube-api-access-rp4wk\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.552204 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:56.552204 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-systemd\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.552204 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-lib-modules\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.552354 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5924\" (UniqueName: \"kubernetes.io/projected/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-kube-api-access-d5924\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.552354 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-systemd-units\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552354 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-env-overrides\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552354 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysctl-d\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.552354 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysctl-conf\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-tuned\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-kubelet\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-slash\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-cni-netd\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysconfig\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.552601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-run\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-systemd\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-ovn\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-hosts-file\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2xz\" (UniqueName: \"kubernetes.io/projected/d9cb1d07-086d-48a2-92a7-f8061493cead-kube-api-access-fg2xz\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-modprobe-d\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-node-log\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-ovnkube-config\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgn7d\" (UniqueName: \"kubernetes.io/projected/3e5babbe-f169-454f-90ad-862aee34194e-kube-api-access-qgn7d\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-run-netns\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-host\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.552968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-tmp\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.553047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e5babbe-f169-454f-90ad-862aee34194e-ovn-node-metrics-cert\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-var-lib-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-kubernetes\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.553699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-var-lib-kubelet\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.553699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-etc-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-sys\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.553699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-log-socket\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.553699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-cni-bin\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.554120 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.553981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dh52p\"" Sep 10 20:56:56.554307 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.554283 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 20:56:56.555093 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.554486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.555405 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.555384 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 20:56:56.556243 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.556170 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 20:56:56.556455 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.556428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:56:56.556579 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.556544 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:56:56.560257 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.560239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.560351 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.560290 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:56:56.563004 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.562981 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.563820 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.563803 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dddgs\"" Sep 10 20:56:56.563927 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.563904 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 20:56:56.564135 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.564116 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 20:56:56.564220 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.564156 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 20:56:56.564355 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.564335 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 20:56:56.564425 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.564364 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 20:56:56.564550 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.564529 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 20:56:56.565484 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.565466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.567878 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.567861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.576064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.576041 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 20:56:56.579343 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.578857 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 20:56:56.579343 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.579144 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 20:56:56.579853 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.579695 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c4mkz\"" Sep 10 20:56:56.579853 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.579759 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 20:56:56.580067 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.580037 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fq9x\"" Sep 10 20:56:56.580406 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.580387 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 20:56:56.580570 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.580540 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 20:56:56.580570 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.580559 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 20:56:56.580721 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.580389 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cbr4f\"" Sep 10 20:56:56.609633 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.609607 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d492q" Sep 10 20:56:56.621216 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.621190 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d492q" Sep 10 20:56:56.637134 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.637087 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 20:56:56.654276 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-node-log\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.654384 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-cni-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.654384 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-daemon-config\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.654384 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-etc-kubernetes\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.654533 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654382 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-node-log\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.654533 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcb8c\" (UniqueName: \"kubernetes.io/projected/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-kube-api-access-fcb8c\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.654533 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b62fe6e0-e353-446e-af6f-89c408244412-serviceca\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.654533 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-tmp\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.654533 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e5babbe-f169-454f-90ad-862aee34194e-ovn-node-metrics-cert\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.654812 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:56:56.654812 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654730 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 20:56:56.654812 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-conf-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.654812 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwzj\" (UniqueName: \"kubernetes.io/projected/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-kube-api-access-6hwzj\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.655038 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-var-lib-kubelet\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.655038 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-cnibin\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.655038 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654931 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-var-lib-kubelet\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.655038 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.654969 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-cni-bin\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.655038 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-hostroot\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.655038 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-host-slash\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-sys\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-sys\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-log-socket\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-log-socket\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bedad80-3589-4b87-a28a-47927ae56347-agent-certs\") pod \"konnectivity-agent-gfcs9\" (UID: \"1bedad80-3589-4b87-a28a-47927ae56347\") " pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-k8s-cni-cncf-io\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-kubelet\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b62fe6e0-e353-446e-af6f-89c408244412-host\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-ovnkube-script-lib\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-socket-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.655334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655301 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-sys-fs\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.655667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:56.655667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-lib-modules\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.655667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5924\" (UniqueName: \"kubernetes.io/projected/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-kube-api-access-d5924\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.655667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-env-overrides\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.655667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-cni-binary-copy\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.655667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-tuned\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.655667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-kubelet\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.655872 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.655872 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-cni-netd\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.655872 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysconfig\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.656006 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655890 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-kubelet\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656006 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-systemd\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656006 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-ovnkube-script-lib\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656006 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655871 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-lib-modules\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.656006 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.655955 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:56:56.656006 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.655993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-ovn\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656189 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.656018 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs podName:d9cb1d07-086d-48a2-92a7-f8061493cead nodeName:}" failed. No retries permitted until 2026-09-10 20:56:57.155989092 +0000 UTC m=+2.974563222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs") pod "network-metrics-daemon-kflkv" (UID: "d9cb1d07-086d-48a2-92a7-f8061493cead") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:56:56.656189 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-multus-certs\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.656189 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.656189 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-hosts-file\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.656189 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-ovnkube-config\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656189 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysconfig\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.656189 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-netns\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.656563 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-ovn\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656563 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-systemd\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656563 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.656563 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-hosts-file\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.656563 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656414 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-cni-netd\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656563 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c4c92c2-36db-4308-b268-f01026044047-kubelet-config\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.656825 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656825 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656659 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-env-overrides\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.656825 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.656815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e5babbe-f169-454f-90ad-862aee34194e-ovnkube-config\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.657147 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-run-netns\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.657230 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-host\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.657230 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-var-lib-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.657230 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-kubernetes\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.657393 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-etc-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.657393 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657264 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.657393 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.657393 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657360 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-cni-bin\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.657393 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-system-cni-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.657644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-registration-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.657644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657456 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-system-cni-dir\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.657644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-tmp-dir\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.657644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bedad80-3589-4b87-a28a-47927ae56347-konnectivity-ca\") pod \"konnectivity-agent-gfcs9\" (UID: \"1bedad80-3589-4b87-a28a-47927ae56347\") " pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.657644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-cni-multus\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.657644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rwq\" (UniqueName: \"kubernetes.io/projected/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-kube-api-access-w5rwq\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwpc9\" (UniqueName: \"kubernetes.io/projected/b62fe6e0-e353-446e-af6f-89c408244412-kube-api-access-lwpc9\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c4c92c2-36db-4308-b268-f01026044047-dbus\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgn7d\" (UniqueName: \"kubernetes.io/projected/3e5babbe-f169-454f-90ad-862aee34194e-kube-api-access-qgn7d\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-cni-binary-copy\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4wk\" (UniqueName: \"kubernetes.io/projected/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-kube-api-access-rp4wk\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-systemd\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-systemd-units\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-cnibin\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-iptables-alerter-script\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysctl-d\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.657968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysctl-conf\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.657982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-slash\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-run\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-socket-dir-parent\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-device-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htt5b\" (UniqueName: \"kubernetes.io/projected/544862f7-a4c6-4753-9932-22668843f699-kube-api-access-htt5b\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658269 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2xz\" (UniqueName: \"kubernetes.io/projected/d9cb1d07-086d-48a2-92a7-f8061493cead-kube-api-access-fg2xz\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-modprobe-d\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-modprobe-d\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658490 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658522 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-host\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-run-netns\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-var-lib-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.658622 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-kubernetes\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-etc-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-cni-bin\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.658996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-tmp-dir\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysctl-d\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-sysctl-conf\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-host-slash\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-systemd\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-run\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-systemd-units\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5babbe-f169-454f-90ad-862aee34194e-run-openvswitch\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.659394 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659373 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-etc-tuned\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.659986 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659890 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e5babbe-f169-454f-90ad-862aee34194e-ovn-node-metrics-cert\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.659986 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.659954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-tmp\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.675532 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.675441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5924\" (UniqueName: \"kubernetes.io/projected/5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc-kube-api-access-d5924\") pod \"tuned-2nl69\" (UID: \"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc\") " pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.680173 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.680146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgn7d\" (UniqueName: \"kubernetes.io/projected/3e5babbe-f169-454f-90ad-862aee34194e-kube-api-access-qgn7d\") pod \"ovnkube-node-xkwq7\" (UID: \"3e5babbe-f169-454f-90ad-862aee34194e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.689644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.689615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4wk\" (UniqueName: \"kubernetes.io/projected/b7b16656-f7d0-4f2e-9326-fa2dda31ff23-kube-api-access-rp4wk\") pod \"node-resolver-tc7ff\" (UID: \"b7b16656-f7d0-4f2e-9326-fa2dda31ff23\") " pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.692152 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.692111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2xz\" (UniqueName: \"kubernetes.io/projected/d9cb1d07-086d-48a2-92a7-f8061493cead-kube-api-access-fg2xz\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:56.759053 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-cni-binary-copy\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.759053 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-multus-certs\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-netns\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c4c92c2-36db-4308-b268-f01026044047-kubelet-config\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-system-cni-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-registration-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-system-cni-dir\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bedad80-3589-4b87-a28a-47927ae56347-konnectivity-ca\") pod \"konnectivity-agent-gfcs9\" (UID: \"1bedad80-3589-4b87-a28a-47927ae56347\") " pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-cni-multus\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rwq\" (UniqueName: \"kubernetes.io/projected/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-kube-api-access-w5rwq\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.759284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwpc9\" (UniqueName: \"kubernetes.io/projected/b62fe6e0-e353-446e-af6f-89c408244412-kube-api-access-lwpc9\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c4c92c2-36db-4308-b268-f01026044047-dbus\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-cni-binary-copy\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-cnibin\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-iptables-alerter-script\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-socket-dir-parent\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-device-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htt5b\" (UniqueName: \"kubernetes.io/projected/544862f7-a4c6-4753-9932-22668843f699-kube-api-access-htt5b\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-cni-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-daemon-config\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-etc-kubernetes\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcb8c\" (UniqueName: \"kubernetes.io/projected/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-kube-api-access-fcb8c\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b62fe6e0-e353-446e-af6f-89c408244412-serviceca\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-conf-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.759708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwzj\" (UniqueName: \"kubernetes.io/projected/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-kube-api-access-6hwzj\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-cnibin\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-cni-bin\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-hostroot\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-host-slash\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bedad80-3589-4b87-a28a-47927ae56347-agent-certs\") pod \"konnectivity-agent-gfcs9\" (UID: \"1bedad80-3589-4b87-a28a-47927ae56347\") " pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-k8s-cni-cncf-io\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-kubelet\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759683 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b62fe6e0-e353-446e-af6f-89c408244412-host\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-socket-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-cni-binary-copy\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759754 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-multus-certs\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-sys-fs\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-sys-fs\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-device-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.759902 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-netns\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760222 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.760050 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.760093 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret podName:8c4c92c2-36db-4308-b268-f01026044047 nodeName:}" failed. No retries permitted until 2026-09-10 20:56:57.260080291 +0000 UTC m=+3.078654390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret") pod "global-pull-secret-syncer-jsfgg" (UID: "8c4c92c2-36db-4308-b268-f01026044047") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c4c92c2-36db-4308-b268-f01026044047-kubelet-config\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-conf-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-run-k8s-cni-cncf-io\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760379 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-tuning-conf-dir\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-kubelet\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.760797 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760447 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-system-cni-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760908 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-daemon-config\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.760976 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-etc-kubernetes\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761113 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/544862f7-a4c6-4753-9932-22668843f699-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-registration-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-cni-bin\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-system-cni-dir\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-hostroot\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761419 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-cnibin\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b62fe6e0-e353-446e-af6f-89c408244412-serviceca\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-cni-dir\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-host-slash\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b62fe6e0-e353-446e-af6f-89c408244412-host\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-socket-dir\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761601 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544862f7-a4c6-4753-9932-22668843f699-cnibin\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761690 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c4c92c2-36db-4308-b268-f01026044047-dbus\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761735 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-host-var-lib-cni-multus\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-iptables-alerter-script\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761799 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-cni-binary-copy\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.761868 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.761811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-multus-socket-dir-parent\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.762438 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.762394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bedad80-3589-4b87-a28a-47927ae56347-konnectivity-ca\") pod \"konnectivity-agent-gfcs9\" (UID: \"1bedad80-3589-4b87-a28a-47927ae56347\") " pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.763400 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.763372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bedad80-3589-4b87-a28a-47927ae56347-agent-certs\") pod \"konnectivity-agent-gfcs9\" (UID: \"1bedad80-3589-4b87-a28a-47927ae56347\") " pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.783681 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.783656 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:56:56.783681 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.783688 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:56:56.783903 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.783703 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2l4j5 for pod openshift-network-diagnostics/network-check-target-6rlj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:56:56.783903 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:56.783770 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5 podName:0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5 nodeName:}" failed. No retries permitted until 2026-09-10 20:56:57.28375225 +0000 UTC m=+3.102326369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2l4j5" (UniqueName: "kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5") pod "network-check-target-6rlj6" (UID: "0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:56:56.786800 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.786749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwzj\" (UniqueName: \"kubernetes.io/projected/bd9a8939-99f6-4a8d-b8ac-47c5897f2aad-kube-api-access-6hwzj\") pod \"iptables-alerter-dnbmd\" (UID: \"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad\") " pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:56.787591 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.787566 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htt5b\" (UniqueName: \"kubernetes.io/projected/544862f7-a4c6-4753-9932-22668843f699-kube-api-access-htt5b\") pod \"multus-additional-cni-plugins-82h58\" (UID: \"544862f7-a4c6-4753-9932-22668843f699\") " pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.788241 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.788207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rwq\" (UniqueName: \"kubernetes.io/projected/ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b-kube-api-access-w5rwq\") pod \"aws-ebs-csi-driver-node-ghtzz\" (UID: \"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.788503 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.788485 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwpc9\" (UniqueName: \"kubernetes.io/projected/b62fe6e0-e353-446e-af6f-89c408244412-kube-api-access-lwpc9\") pod \"node-ca-pmgvp\" (UID: \"b62fe6e0-e353-446e-af6f-89c408244412\") " pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.788712 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.788691 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcb8c\" (UniqueName: \"kubernetes.io/projected/9ca6eb1f-39ca-4e13-9451-c2e2ac879135-kube-api-access-fcb8c\") pod \"multus-t57dn\" (UID: \"9ca6eb1f-39ca-4e13-9451-c2e2ac879135\") " pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.840663 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.840629 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:56:56.843567 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.843548 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tc7ff" Sep 10 20:56:56.862711 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.862685 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nl69" Sep 10 20:56:56.871600 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.871582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:56:56.879169 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.879146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pmgvp" Sep 10 20:56:56.887021 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.887003 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:56:56.894611 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.894593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-82h58" Sep 10 20:56:56.905408 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.905390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t57dn" Sep 10 20:56:56.913524 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.913503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" Sep 10 20:56:56.922328 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:56.922310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dnbmd" Sep 10 20:56:57.163060 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:57.162990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:57.163196 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.163117 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:56:57.163196 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.163178 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs podName:d9cb1d07-086d-48a2-92a7-f8061493cead nodeName:}" failed. No retries permitted until 2026-09-10 20:56:58.163160811 +0000 UTC m=+3.981734911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs") pod "network-metrics-daemon-kflkv" (UID: "d9cb1d07-086d-48a2-92a7-f8061493cead") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:56:57.264034 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:57.264000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:57.264222 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.264152 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:56:57.264222 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.264217 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret podName:8c4c92c2-36db-4308-b268-f01026044047 nodeName:}" failed. No retries permitted until 2026-09-10 20:56:58.264199643 +0000 UTC m=+4.082773762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret") pod "global-pull-secret-syncer-jsfgg" (UID: "8c4c92c2-36db-4308-b268-f01026044047") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:56:57.365149 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:57.365110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:56:57.365302 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.365282 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:56:57.365355 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.365310 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:56:57.365355 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.365323 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2l4j5 for pod openshift-network-diagnostics/network-check-target-6rlj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:56:57.365439 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.365407 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5 podName:0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5 nodeName:}" failed. No retries permitted until 2026-09-10 20:56:58.365386984 +0000 UTC m=+4.183961154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l4j5" (UniqueName: "kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5") pod "network-check-target-6rlj6" (UID: "0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:56:57.624739 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:57.624649 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:51:57 +0000 UTC" deadline="2028-02-19 15:21:26.526630832 +0000 UTC" Sep 10 20:56:57.624739 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:57.624688 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12642h24m28.901947641s" Sep 10 20:56:57.645061 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:57.645034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:57.645207 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:57.645185 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:56:57.839108 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:57.837464 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:56:57.867760 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:57.867690 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62fe6e0_e353_446e_af6f_89c408244412.slice/crio-01e391b9974fb2035ee79609de8788eb36e415795ccf1af8dfd0b6b37fc24648 WatchSource:0}: Error finding container 01e391b9974fb2035ee79609de8788eb36e415795ccf1af8dfd0b6b37fc24648: Status 404 returned error can't find the container with id 01e391b9974fb2035ee79609de8788eb36e415795ccf1af8dfd0b6b37fc24648 Sep 10 20:56:57.870290 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:57.870258 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544862f7_a4c6_4753_9932_22668843f699.slice/crio-41e7bb7894e63e8f895daeb2b2a22d891e9e3cfdda839751d0e94c5e3542ae6d WatchSource:0}: Error finding container 41e7bb7894e63e8f895daeb2b2a22d891e9e3cfdda839751d0e94c5e3542ae6d: Status 404 returned error can't find the container with id 41e7bb7894e63e8f895daeb2b2a22d891e9e3cfdda839751d0e94c5e3542ae6d Sep 10 20:56:57.870628 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:57.870600 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bedad80_3589_4b87_a28a_47927ae56347.slice/crio-93fea4c48f6ef2312ee9c8f328dfd34f535efe69ce14d89907f8c1fbe4f5c11c WatchSource:0}: Error finding container 93fea4c48f6ef2312ee9c8f328dfd34f535efe69ce14d89907f8c1fbe4f5c11c: Status 404 returned error can't find the container with id 93fea4c48f6ef2312ee9c8f328dfd34f535efe69ce14d89907f8c1fbe4f5c11c Sep 10 20:56:57.874020 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:57.873985 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9a8939_99f6_4a8d_b8ac_47c5897f2aad.slice/crio-490d1959438da47590b12c9636335b61a76594ddd21679895408cb5ce54a307c WatchSource:0}: Error finding container 490d1959438da47590b12c9636335b61a76594ddd21679895408cb5ce54a307c: Status 404 returned error can't find the container with id 490d1959438da47590b12c9636335b61a76594ddd21679895408cb5ce54a307c Sep 10 20:56:57.875365 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:57.875296 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca6eb1f_39ca_4e13_9451_c2e2ac879135.slice/crio-0ab5f8aaca11ac92c40cb8c0cc27dce4b6feb54bb044e3518a8fb91d9be47be3 WatchSource:0}: Error finding container 0ab5f8aaca11ac92c40cb8c0cc27dce4b6feb54bb044e3518a8fb91d9be47be3: Status 404 returned error can't find the container with id 0ab5f8aaca11ac92c40cb8c0cc27dce4b6feb54bb044e3518a8fb91d9be47be3 Sep 10 20:56:57.876641 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:57.876602 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b16656_f7d0_4f2e_9326_fa2dda31ff23.slice/crio-2db46fcdf6e079ff3bdc8b93d048744ced03520990e90a118fa91db62c89b237 WatchSource:0}: Error finding container 2db46fcdf6e079ff3bdc8b93d048744ced03520990e90a118fa91db62c89b237: Status 404 returned error can't find the container with id 2db46fcdf6e079ff3bdc8b93d048744ced03520990e90a118fa91db62c89b237 Sep 10 20:56:57.877971 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:56:57.877951 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d13ac2f_dd2f_4a2d_9ff2_e10f9f4f71cc.slice/crio-d886bbe2c19b69fc9f757b8a936d2586a70a03e16a84e92f71b9d3df17f4a8a0 WatchSource:0}: Error finding container d886bbe2c19b69fc9f757b8a936d2586a70a03e16a84e92f71b9d3df17f4a8a0: Status 404 returned error can't find the container with id d886bbe2c19b69fc9f757b8a936d2586a70a03e16a84e92f71b9d3df17f4a8a0 Sep 10 20:56:58.171595 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.171380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:58.171695 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.171518 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:56:58.171695 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.171683 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs podName:d9cb1d07-086d-48a2-92a7-f8061493cead nodeName:}" failed. No retries permitted until 2026-09-10 20:57:00.171663512 +0000 UTC m=+5.990237613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs") pod "network-metrics-daemon-kflkv" (UID: "d9cb1d07-086d-48a2-92a7-f8061493cead") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:56:58.272263 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.272213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:58.272431 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.272364 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:56:58.272431 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.272416 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret podName:8c4c92c2-36db-4308-b268-f01026044047 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:00.272401743 +0000 UTC m=+6.090975857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret") pod "global-pull-secret-syncer-jsfgg" (UID: "8c4c92c2-36db-4308-b268-f01026044047") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:56:58.373068 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.373024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:56:58.373298 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.373275 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:56:58.373380 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.373304 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:56:58.373380 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.373319 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2l4j5 for pod openshift-network-diagnostics/network-check-target-6rlj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:56:58.373490 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.373387 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5 podName:0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:00.373365318 +0000 UTC m=+6.191939435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l4j5" (UniqueName: "kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5") pod "network-check-target-6rlj6" (UID: "0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:56:58.509047 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.508964 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 20:56:58.625795 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.625720 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:51:57 +0000 UTC" deadline="2028-05-14 00:37:28.562580697 +0000 UTC" Sep 10 20:56:58.625795 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.625764 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14667h40m29.936819202s" Sep 10 20:56:58.647996 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.647232 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:56:58.647996 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.647366 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:56:58.647996 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.647760 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:56:58.647996 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:58.647853 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:56:58.666204 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.666170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" event={"ID":"0f953879e24b63da8b00052cd2968abb","Type":"ContainerStarted","Data":"32e7734d3e43194674e2704027a716e53c337769f1f578f5eda35229e20996d5"} Sep 10 20:56:58.677708 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.677618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" event={"ID":"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b","Type":"ContainerStarted","Data":"fd436809bc66c230dc7e82b6fded9c48551b53a58fb565575c93e545da7b9a39"} Sep 10 20:56:58.692165 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.692098 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nl69" event={"ID":"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc","Type":"ContainerStarted","Data":"d886bbe2c19b69fc9f757b8a936d2586a70a03e16a84e92f71b9d3df17f4a8a0"} Sep 10 20:56:58.699901 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.699857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t57dn" event={"ID":"9ca6eb1f-39ca-4e13-9451-c2e2ac879135","Type":"ContainerStarted","Data":"0ab5f8aaca11ac92c40cb8c0cc27dce4b6feb54bb044e3518a8fb91d9be47be3"} Sep 10 20:56:58.708630 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.708601 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gfcs9" event={"ID":"1bedad80-3589-4b87-a28a-47927ae56347","Type":"ContainerStarted","Data":"93fea4c48f6ef2312ee9c8f328dfd34f535efe69ce14d89907f8c1fbe4f5c11c"} Sep 10 20:56:58.724851 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.724803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerStarted","Data":"41e7bb7894e63e8f895daeb2b2a22d891e9e3cfdda839751d0e94c5e3542ae6d"} Sep 10 20:56:58.742968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.742936 2574 generic.go:356] "Generic (PLEG): container finished" podID="a062f58674a98f4e1b68c4c7aec1dbff" containerID="51fa7fdf74ca3ae3169743c194537973461e789d586bd04c2b9d707616187de2" exitCode=0 Sep 10 20:56:58.743284 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.743260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" event={"ID":"a062f58674a98f4e1b68c4c7aec1dbff","Type":"ContainerDied","Data":"51fa7fdf74ca3ae3169743c194537973461e789d586bd04c2b9d707616187de2"} Sep 10 20:56:58.753953 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.753828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"b376bc9a78ed0c71f12f59e401d5d37a0bc73daf7d09a51b331f1d8134398674"} Sep 10 20:56:58.772588 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.772540 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-255.ec2.internal" podStartSLOduration=2.77251583 podStartE2EDuration="2.77251583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:56:58.694232489 +0000 UTC m=+4.512806615" watchObservedRunningTime="2026-09-10 20:56:58.77251583 +0000 UTC m=+4.591089974" Sep 10 20:56:58.778368 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.778308 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tc7ff" event={"ID":"b7b16656-f7d0-4f2e-9326-fa2dda31ff23","Type":"ContainerStarted","Data":"2db46fcdf6e079ff3bdc8b93d048744ced03520990e90a118fa91db62c89b237"} Sep 10 20:56:58.781161 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.781094 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dnbmd" event={"ID":"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad","Type":"ContainerStarted","Data":"490d1959438da47590b12c9636335b61a76594ddd21679895408cb5ce54a307c"} Sep 10 20:56:58.783369 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:58.783192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pmgvp" event={"ID":"b62fe6e0-e353-446e-af6f-89c408244412","Type":"ContainerStarted","Data":"01e391b9974fb2035ee79609de8788eb36e415795ccf1af8dfd0b6b37fc24648"} Sep 10 20:56:59.644428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:59.644394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:56:59.644987 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:56:59.644545 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:56:59.790528 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:56:59.790491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" event={"ID":"a062f58674a98f4e1b68c4c7aec1dbff","Type":"ContainerStarted","Data":"2d8c0135a791924ad877caa269358b89bc83edd5d253b12aa099a036ab66f00e"} Sep 10 20:57:00.189496 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:00.188797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:00.189496 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.189041 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:00.189496 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.189110 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs podName:d9cb1d07-086d-48a2-92a7-f8061493cead nodeName:}" failed. No retries permitted until 2026-09-10 20:57:04.189089321 +0000 UTC m=+10.007663429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs") pod "network-metrics-daemon-kflkv" (UID: "d9cb1d07-086d-48a2-92a7-f8061493cead") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:00.290832 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:00.290196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:00.290832 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.290408 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:00.290832 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.290474 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret podName:8c4c92c2-36db-4308-b268-f01026044047 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:04.290455049 +0000 UTC m=+10.109029155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret") pod "global-pull-secret-syncer-jsfgg" (UID: "8c4c92c2-36db-4308-b268-f01026044047") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:00.390949 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:00.390691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:00.391853 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.391150 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:57:00.391853 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.391183 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:57:00.391853 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.391197 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2l4j5 for pod openshift-network-diagnostics/network-check-target-6rlj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:00.391853 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.391268 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5 podName:0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:04.391248991 +0000 UTC m=+10.209823097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l4j5" (UniqueName: "kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5") pod "network-check-target-6rlj6" (UID: "0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:00.645561 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:00.644969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:00.645561 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.645109 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:00.646450 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:00.646199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:00.646450 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:00.646324 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:01.644421 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:01.644380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:01.644638 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:01.644532 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:02.646017 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:02.645985 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:02.646489 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:02.646115 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:02.647294 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:02.647270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:02.647395 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:02.647367 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:03.644881 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:03.644773 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:03.645054 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:03.644931 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:04.221060 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:04.221015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:04.221511 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.221190 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:04.221511 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.221269 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs podName:d9cb1d07-086d-48a2-92a7-f8061493cead nodeName:}" failed. No retries permitted until 2026-09-10 20:57:12.221248206 +0000 UTC m=+18.039822323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs") pod "network-metrics-daemon-kflkv" (UID: "d9cb1d07-086d-48a2-92a7-f8061493cead") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:04.322468 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:04.322419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:04.322634 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.322594 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:04.322704 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.322679 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret podName:8c4c92c2-36db-4308-b268-f01026044047 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:12.322659411 +0000 UTC m=+18.141233527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret") pod "global-pull-secret-syncer-jsfgg" (UID: "8c4c92c2-36db-4308-b268-f01026044047") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:04.424079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:04.423389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:04.424079 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.423588 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:57:04.424079 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.423611 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:57:04.424079 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.423624 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2l4j5 for pod openshift-network-diagnostics/network-check-target-6rlj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:04.424079 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.423697 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5 podName:0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:12.423666059 +0000 UTC m=+18.242240162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l4j5" (UniqueName: "kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5") pod "network-check-target-6rlj6" (UID: "0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:04.646416 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:04.645870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:04.646416 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.646009 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:04.646416 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:04.646062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:04.646416 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:04.646207 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:05.644737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:05.644695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:05.645190 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:05.644862 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:06.646512 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:06.646006 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:06.646512 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:06.646013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:06.646512 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:06.646153 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:06.646512 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:06.646239 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:07.644427 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:07.644396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:07.644614 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:07.644534 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:08.644201 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:08.644173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:08.644627 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:08.644296 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:08.644627 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:08.644340 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:08.644627 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:08.644424 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:09.644606 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:09.644574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:09.645063 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:09.644720 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:10.644431 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:10.644396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:10.644645 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:10.644405 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:10.644645 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:10.644515 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:10.644645 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:10.644611 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:11.644427 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:11.644394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:11.644623 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:11.644535 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:12.284223 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:12.284187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:12.284620 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.284359 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:12.284620 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.284423 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs podName:d9cb1d07-086d-48a2-92a7-f8061493cead nodeName:}" failed. No retries permitted until 2026-09-10 20:57:28.284407356 +0000 UTC m=+34.102981473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs") pod "network-metrics-daemon-kflkv" (UID: "d9cb1d07-086d-48a2-92a7-f8061493cead") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:12.385226 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:12.385182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:12.385399 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.385353 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:12.385463 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.385422 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret podName:8c4c92c2-36db-4308-b268-f01026044047 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:28.385404143 +0000 UTC m=+34.203978246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret") pod "global-pull-secret-syncer-jsfgg" (UID: "8c4c92c2-36db-4308-b268-f01026044047") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:12.486388 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:12.486354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:12.486545 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.486513 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:57:12.486545 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.486535 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:57:12.486545 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.486547 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2l4j5 for pod openshift-network-diagnostics/network-check-target-6rlj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:12.486703 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.486604 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5 podName:0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:28.486585238 +0000 UTC m=+34.305159355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l4j5" (UniqueName: "kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5") pod "network-check-target-6rlj6" (UID: "0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:12.644590 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:12.644520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:12.644735 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:12.644520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:12.644735 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.644648 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:12.644735 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:12.644726 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:13.644775 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:13.644738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:13.645188 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:13.644876 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:14.645914 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:14.645884 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:14.646472 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:14.645953 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:14.646602 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:14.646488 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:14.646602 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:14.646571 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:15.644383 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:15.644346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:15.644552 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:15.644479 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:16.644320 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:16.644273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:16.644907 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:16.644327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:16.644907 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:16.644420 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:16.644907 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:16.644540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:17.645171 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.644857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:17.645981 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:17.645289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:17.841393 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.841358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gfcs9" event={"ID":"1bedad80-3589-4b87-a28a-47927ae56347","Type":"ContainerStarted","Data":"bb5fcb5c2a49b90806b5c44cc2dd3af7f2ef9a6e962e62710d1bfdb4a1e9214d"} Sep 10 20:57:17.842916 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.842892 2574 generic.go:356] "Generic (PLEG): container finished" podID="544862f7-a4c6-4753-9932-22668843f699" containerID="97e487c96b308e12b87915c3b2047406324af2f79a97096be2379af36b2007d8" exitCode=0 Sep 10 20:57:17.843044 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.842929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerDied","Data":"97e487c96b308e12b87915c3b2047406324af2f79a97096be2379af36b2007d8"} Sep 10 20:57:17.846426 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.846402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"df2528f25b38b001c24f55545d4892fc2548ddebe9963cb1ba7b19f9fdfdf23f"} Sep 10 20:57:17.846551 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.846480 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"504c46d1dcadff749b2d69d4c1cc0f5b99f7e188552b12a932532498a746501e"} Sep 10 20:57:17.846551 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.846529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"3b68791259a8686de689ea836bbbd31090189310a3338c86422dfde6b7a89c45"} Sep 10 20:57:17.846669 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.846575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"6602b574dbcd0c62a3cc26ae4e22b81f897b926bc63071b4db5cab2d50b96f4c"} Sep 10 20:57:17.846669 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.846626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"e910285a12afdd536ee82c7f058f3469efb092f5ef9d71056994c85f5d7b4206"} Sep 10 20:57:17.846766 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.846674 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"8091ba7c8526ba72470330b915b5bab5be0cde96ac11dfffdbf141a56a4ab979"} Sep 10 20:57:17.847940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.847919 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tc7ff" event={"ID":"b7b16656-f7d0-4f2e-9326-fa2dda31ff23","Type":"ContainerStarted","Data":"6bb1811654bd57fb83bd2bbece68a37d18bfae0234a80f4255b65bfc7db7cdf4"} Sep 10 20:57:17.849532 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.849512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pmgvp" event={"ID":"b62fe6e0-e353-446e-af6f-89c408244412","Type":"ContainerStarted","Data":"fa059ea711329b8ce74df7352cb58713e868b88af76988c57f2d8a89e23c36cb"} Sep 10 20:57:17.851524 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.851499 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" event={"ID":"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b","Type":"ContainerStarted","Data":"ac124c7f70a14b07d4ae64d591e52c6960f8f8d7bb83a840f87024e9da3e8597"} Sep 10 20:57:17.853077 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.853053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nl69" event={"ID":"5d13ac2f-dd2f-4a2d-9ff2-e10f9f4f71cc","Type":"ContainerStarted","Data":"c7bb004082eac9dd180a7e5b6ff17819ce9f08a23501ab578f074f3531fb6044"} Sep 10 20:57:17.854637 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.854608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t57dn" event={"ID":"9ca6eb1f-39ca-4e13-9451-c2e2ac879135","Type":"ContainerStarted","Data":"9abf846c78a66b5f3fdd126841d72cf1c98ef2617d7bb2135e633399dbd60c9b"} Sep 10 20:57:17.861321 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.861286 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-255.ec2.internal" podStartSLOduration=21.861274571 podStartE2EDuration="21.861274571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:56:59.816120667 +0000 UTC m=+5.634694794" watchObservedRunningTime="2026-09-10 20:57:17.861274571 +0000 UTC m=+23.679848692" Sep 10 20:57:17.883718 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.883676 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gfcs9" podStartSLOduration=7.855611142 podStartE2EDuration="22.88366294s" totalImagesPullingTime="15.028051798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.861066497 +0000 UTC m=+23.679640618" watchObservedRunningTime="2026-09-10 20:57:17.88366294 +0000 UTC m=+23.702237061" Sep 10 20:57:17.929752 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.929677 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2nl69" podStartSLOduration=4.016643693 podStartE2EDuration="22.929653043s" totalImagesPullingTime="18.91300935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.928477846 +0000 UTC m=+23.747051966" watchObservedRunningTime="2026-09-10 20:57:17.929653043 +0000 UTC m=+23.748227164" Sep 10 20:57:17.930526 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.930491 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tc7ff" podStartSLOduration=4.031453114 podStartE2EDuration="22.930482335s" totalImagesPullingTime="18.899029221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.907323417 +0000 UTC m=+23.725897539" watchObservedRunningTime="2026-09-10 20:57:17.930482335 +0000 UTC m=+23.749056458" Sep 10 20:57:17.945523 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.945486 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pmgvp" podStartSLOduration=4.037132785 podStartE2EDuration="22.94547362s" totalImagesPullingTime="18.908340835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.945282398 +0000 UTC m=+23.763856550" watchObservedRunningTime="2026-09-10 20:57:17.94547362 +0000 UTC m=+23.764047790" Sep 10 20:57:17.969900 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:17.969850 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t57dn" podStartSLOduration=3.006029133 podStartE2EDuration="21.969835084s" totalImagesPullingTime="18.963805951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.969258209 +0000 UTC m=+23.787832334" watchObservedRunningTime="2026-09-10 20:57:17.969835084 +0000 UTC m=+23.788409203" Sep 10 20:57:18.290658 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.290638 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 20:57:18.612131 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.612009 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:57:18.290654453Z","UUID":"c7d64039-3e6c-4f08-957d-901cda74ff63","Handler":null,"Name":"","Endpoint":""} Sep 10 20:57:18.613342 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.613324 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 20:57:18.613438 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.613353 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 20:57:18.644324 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.644302 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:18.644464 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:18.644398 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:18.644524 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.644490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:18.644615 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:18.644595 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:18.860896 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.860857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dnbmd" event={"ID":"bd9a8939-99f6-4a8d-b8ac-47c5897f2aad","Type":"ContainerStarted","Data":"78eb221a2b7c1d9c8b9f968e9307bb135fa28c0ce3d3665a26d80509a10055a8"} Sep 10 20:57:18.863838 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:18.863751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" event={"ID":"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b","Type":"ContainerStarted","Data":"134fc3c3758be3616a8ebff4669a5eafba9f2281c1bc252b24f34ba7eb307acc"} Sep 10 20:57:19.144227 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.144130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:57:19.144709 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.144691 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:57:19.165627 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.165583 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dnbmd" podStartSLOduration=4.261729816 podStartE2EDuration="23.165564987s" totalImagesPullingTime="18.903835171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:18.882211361 +0000 UTC m=+24.700785507" watchObservedRunningTime="2026-09-10 20:57:19.165564987 +0000 UTC m=+24.984139109" Sep 10 20:57:19.644377 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.644349 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:19.644517 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:19.644455 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:19.871349 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.871299 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"53487905ac4030fa77bab4d10429d1b22a04f178a6a5fbdd9b39e2c414096603"} Sep 10 20:57:19.873500 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.873470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" event={"ID":"ce9bbdba-6cca-4891-9d4c-0a08ce03fd8b","Type":"ContainerStarted","Data":"982921fe67bd560ae90cedd9f909407d6c991eb2ea63aae451d8f5d855aad780"} Sep 10 20:57:19.874003 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.873845 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:57:19.874355 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.874334 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gfcs9" Sep 10 20:57:19.901045 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:19.900997 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ghtzz" podStartSLOduration=2.667388265 podStartE2EDuration="23.900980733s" totalImagesPullingTime="21.233592468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:19.900946455 +0000 UTC m=+25.719520589" watchObservedRunningTime="2026-09-10 20:57:19.900980733 +0000 UTC m=+25.719554854" Sep 10 20:57:20.644733 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:20.644694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:20.644733 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:20.644722 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:20.644980 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:20.644841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:20.644980 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:20.644890 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:21.644401 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:21.644224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:21.644788 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:21.644510 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:22.644874 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.644701 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:22.645625 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.644756 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:22.645625 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:22.644969 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:22.645625 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:22.645012 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:22.882130 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.882017 2574 generic.go:356] "Generic (PLEG): container finished" podID="544862f7-a4c6-4753-9932-22668843f699" containerID="581670d1bda80383c44f0e46c18f4281e6237b08d5728456cc6060805cb5c4ff" exitCode=0 Sep 10 20:57:22.882130 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.882096 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerDied","Data":"581670d1bda80383c44f0e46c18f4281e6237b08d5728456cc6060805cb5c4ff"} Sep 10 20:57:22.885910 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.885889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" event={"ID":"3e5babbe-f169-454f-90ad-862aee34194e","Type":"ContainerStarted","Data":"50608620c75490a3f3947a6f13468260a6c7a8480b27ce56058ee324baab9b1f"} Sep 10 20:57:22.886167 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.886152 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:57:22.901749 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.901732 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:57:22.951602 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:22.951558 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" podStartSLOduration=8.990282863000001 podStartE2EDuration="27.951545302s" totalImagesPullingTime="18.961262439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:22.951295042 +0000 UTC m=+28.769869164" watchObservedRunningTime="2026-09-10 20:57:22.951545302 +0000 UTC m=+28.770119422" Sep 10 20:57:23.650328 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:23.650291 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:23.650841 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:23.650438 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:23.888364 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:23.888337 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 20:57:23.888748 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:23.888733 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:57:23.903763 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:23.903709 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:57:24.335954 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.335924 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6rlj6"] Sep 10 20:57:24.336117 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.336041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:24.336165 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:24.336141 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:24.339019 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.338996 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jsfgg"] Sep 10 20:57:24.339108 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.339083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:24.339173 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:24.339156 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:24.342090 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.342061 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kflkv"] Sep 10 20:57:24.342256 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.342236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:24.342423 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:24.342395 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:24.893109 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.893074 2574 generic.go:356] "Generic (PLEG): container finished" podID="544862f7-a4c6-4753-9932-22668843f699" containerID="e4df055fe2072bfff37fd91866efa2011a75023a701449c5ff24ad6ff8624ead" exitCode=0 Sep 10 20:57:24.893559 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.893158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerDied","Data":"e4df055fe2072bfff37fd91866efa2011a75023a701449c5ff24ad6ff8624ead"} Sep 10 20:57:24.893559 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.893227 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 20:57:24.936705 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:24.936679 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:57:25.644859 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:25.644803 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:25.645033 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:25.644947 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:25.897938 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:25.897682 2574 generic.go:356] "Generic (PLEG): container finished" podID="544862f7-a4c6-4753-9932-22668843f699" containerID="15b4984c47648c2bc149cd3a985cea61b794b9fe783bf36a6803eda3099ea504" exitCode=0 Sep 10 20:57:25.898320 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:25.897771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerDied","Data":"15b4984c47648c2bc149cd3a985cea61b794b9fe783bf36a6803eda3099ea504"} Sep 10 20:57:26.647024 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:26.646998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:26.647155 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:26.647110 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:26.647203 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:26.647163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:26.647273 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:26.647254 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:27.644653 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:27.644620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:27.645097 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:27.644724 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6rlj6" podUID="0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5" Sep 10 20:57:28.301802 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:28.301767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:28.302053 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.301934 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:28.302053 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.302014 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs podName:d9cb1d07-086d-48a2-92a7-f8061493cead nodeName:}" failed. No retries permitted until 2026-09-10 20:58:00.301997661 +0000 UTC m=+66.120571760 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs") pod "network-metrics-daemon-kflkv" (UID: "d9cb1d07-086d-48a2-92a7-f8061493cead") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 20:57:28.402829 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:28.402776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:28.403012 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.402895 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:28.403012 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.402947 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret podName:8c4c92c2-36db-4308-b268-f01026044047 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:00.402934244 +0000 UTC m=+66.221508342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret") pod "global-pull-secret-syncer-jsfgg" (UID: "8c4c92c2-36db-4308-b268-f01026044047") : object "kube-system"/"original-pull-secret" not registered Sep 10 20:57:28.503224 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:28.503189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:28.503392 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.503368 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 20:57:28.503392 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.503390 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 20:57:28.503477 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.503405 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2l4j5 for pod openshift-network-diagnostics/network-check-target-6rlj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:28.503477 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.503464 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5 podName:0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:00.503446656 +0000 UTC m=+66.322020755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l4j5" (UniqueName: "kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5") pod "network-check-target-6rlj6" (UID: "0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 20:57:28.645128 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:28.645051 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:28.645601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:28.645054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:28.645601 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.645177 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jsfgg" podUID="8c4c92c2-36db-4308-b268-f01026044047" Sep 10 20:57:28.645601 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:28.645235 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kflkv" podUID="d9cb1d07-086d-48a2-92a7-f8061493cead" Sep 10 20:57:29.480456 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.480421 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-255.ec2.internal" event="NodeReady" Sep 10 20:57:29.480625 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.480609 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 20:57:29.529597 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.529556 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b877979b8-6xrtf"] Sep 10 20:57:29.534960 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.534939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.538402 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.538382 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 20:57:29.538514 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.538455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-24mm2\"" Sep 10 20:57:29.538514 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.538504 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 20:57:29.538738 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.538697 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 20:57:29.546852 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.546829 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 20:57:29.564641 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.564573 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b877979b8-6xrtf"] Sep 10 20:57:29.579049 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.579020 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pbkpz"] Sep 10 20:57:29.591582 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.591557 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rzdqm"] Sep 10 20:57:29.591709 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.591683 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.595098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.595074 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw26g\"" Sep 10 20:57:29.595617 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.595596 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 20:57:29.595726 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.595700 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 20:57:29.601786 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.601735 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rzdqm"] Sep 10 20:57:29.601786 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.601772 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbkpz"] Sep 10 20:57:29.601970 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.601879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:29.605663 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.605639 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 20:57:29.605757 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.605734 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 20:57:29.606142 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.606123 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 20:57:29.606234 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.606162 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kgg6r\"" Sep 10 20:57:29.609531 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-image-registry-private-configuration\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.609531 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-certificates\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.609531 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-bound-sa-token\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.609531 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjjd\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-kube-api-access-kkjjd\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.609751 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-ca-trust-extracted\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.609751 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.609751 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-trusted-ca\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.609751 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.609673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-installation-pull-secrets\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.644556 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.644527 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:57:29.649581 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.649559 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gncgs\"" Sep 10 20:57:29.650182 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.650167 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 20:57:29.650272 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.650207 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 20:57:29.710566 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-image-registry-private-configuration\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.710566 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7cv\" (UniqueName: \"kubernetes.io/projected/597fc486-2ab7-49cb-91ae-9d4a37bd174b-kube-api-access-vb7cv\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.710831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-certificates\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.710831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn28t\" (UniqueName: \"kubernetes.io/projected/a91b9b2b-9609-4f4d-9bcd-07e184a8f19d-kube-api-access-jn28t\") pod \"ingress-canary-rzdqm\" (UID: \"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d\") " pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:29.710831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.710831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-bound-sa-token\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.710831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/597fc486-2ab7-49cb-91ae-9d4a37bd174b-config-volume\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.711064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjjd\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-kube-api-access-kkjjd\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.711064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a91b9b2b-9609-4f4d-9bcd-07e184a8f19d-cert\") pod \"ingress-canary-rzdqm\" (UID: \"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d\") " pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:29.711064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-ca-trust-extracted\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.711064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.711064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.710991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-trusted-ca\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.711064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.711037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-installation-pull-secrets\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.711351 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.711066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/597fc486-2ab7-49cb-91ae-9d4a37bd174b-tmp-dir\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.711351 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:29.711126 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 20:57:29.711351 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:29.711141 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b877979b8-6xrtf: secret "image-registry-tls" not found Sep 10 20:57:29.711351 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:29.711192 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls podName:5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:30.211174016 +0000 UTC m=+36.029748119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls") pod "image-registry-b877979b8-6xrtf" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58") : secret "image-registry-tls" not found Sep 10 20:57:29.711351 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.711268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-certificates\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.711591 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.711569 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-ca-trust-extracted\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.711760 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.711731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-trusted-ca\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.715575 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.715550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-installation-pull-secrets\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.715748 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.715579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-image-registry-private-configuration\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.723008 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.722982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-bound-sa-token\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.723720 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.723695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjjd\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-kube-api-access-kkjjd\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:29.812000 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.811967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7cv\" (UniqueName: \"kubernetes.io/projected/597fc486-2ab7-49cb-91ae-9d4a37bd174b-kube-api-access-vb7cv\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.812184 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.812012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn28t\" (UniqueName: \"kubernetes.io/projected/a91b9b2b-9609-4f4d-9bcd-07e184a8f19d-kube-api-access-jn28t\") pod \"ingress-canary-rzdqm\" (UID: \"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d\") " pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:29.812184 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.812035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.812184 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.812055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/597fc486-2ab7-49cb-91ae-9d4a37bd174b-config-volume\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.812184 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.812079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a91b9b2b-9609-4f4d-9bcd-07e184a8f19d-cert\") pod \"ingress-canary-rzdqm\" (UID: \"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d\") " pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:29.812397 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.812189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/597fc486-2ab7-49cb-91ae-9d4a37bd174b-tmp-dir\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.812397 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:29.812202 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:57:29.812397 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:29.812274 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls podName:597fc486-2ab7-49cb-91ae-9d4a37bd174b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:30.312254683 +0000 UTC m=+36.130828802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls") pod "dns-default-pbkpz" (UID: "597fc486-2ab7-49cb-91ae-9d4a37bd174b") : secret "dns-default-metrics-tls" not found Sep 10 20:57:29.812596 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.812578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/597fc486-2ab7-49cb-91ae-9d4a37bd174b-tmp-dir\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.812801 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.812782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/597fc486-2ab7-49cb-91ae-9d4a37bd174b-config-volume\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.815175 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.815112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a91b9b2b-9609-4f4d-9bcd-07e184a8f19d-cert\") pod \"ingress-canary-rzdqm\" (UID: \"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d\") " pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:29.826304 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.826277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7cv\" (UniqueName: \"kubernetes.io/projected/597fc486-2ab7-49cb-91ae-9d4a37bd174b-kube-api-access-vb7cv\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:29.826443 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.826420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn28t\" (UniqueName: \"kubernetes.io/projected/a91b9b2b-9609-4f4d-9bcd-07e184a8f19d-kube-api-access-jn28t\") pod \"ingress-canary-rzdqm\" (UID: \"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d\") " pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:29.912404 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:29.912376 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rzdqm" Sep 10 20:57:30.214689 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:30.214606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:30.214863 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:30.214788 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 20:57:30.214863 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:30.214825 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b877979b8-6xrtf: secret "image-registry-tls" not found Sep 10 20:57:30.214980 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:30.214898 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls podName:5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:31.214876125 +0000 UTC m=+37.033450246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls") pod "image-registry-b877979b8-6xrtf" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58") : secret "image-registry-tls" not found Sep 10 20:57:30.315924 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:30.315865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:30.316109 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:30.316036 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:57:30.316180 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:30.316119 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls podName:597fc486-2ab7-49cb-91ae-9d4a37bd174b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:31.316094613 +0000 UTC m=+37.134668713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls") pod "dns-default-pbkpz" (UID: "597fc486-2ab7-49cb-91ae-9d4a37bd174b") : secret "dns-default-metrics-tls" not found Sep 10 20:57:30.644552 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:30.644512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:57:30.644754 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:30.644512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:57:30.650090 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:30.650065 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zr4bv\"" Sep 10 20:57:30.650637 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:30.650443 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 20:57:30.650718 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:30.650701 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 20:57:31.223587 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:31.223410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:31.223733 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:31.223567 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 20:57:31.223733 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:31.223655 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b877979b8-6xrtf: secret "image-registry-tls" not found Sep 10 20:57:31.223733 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:31.223708 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls podName:5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:33.223694261 +0000 UTC m=+39.042268373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls") pod "image-registry-b877979b8-6xrtf" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58") : secret "image-registry-tls" not found Sep 10 20:57:31.280023 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:31.279984 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rzdqm"] Sep 10 20:57:31.283670 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:57:31.283641 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91b9b2b_9609_4f4d_9bcd_07e184a8f19d.slice/crio-a2ae0bb9e3e79c350b6f7d871e0df26db72ddc4ddfaba2a5ed540c31d8f36af8 WatchSource:0}: Error finding container a2ae0bb9e3e79c350b6f7d871e0df26db72ddc4ddfaba2a5ed540c31d8f36af8: Status 404 returned error can't find the container with id a2ae0bb9e3e79c350b6f7d871e0df26db72ddc4ddfaba2a5ed540c31d8f36af8 Sep 10 20:57:31.324508 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:31.324485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:31.324661 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:31.324645 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:57:31.324729 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:31.324715 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls podName:597fc486-2ab7-49cb-91ae-9d4a37bd174b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:33.324695427 +0000 UTC m=+39.143269543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls") pod "dns-default-pbkpz" (UID: "597fc486-2ab7-49cb-91ae-9d4a37bd174b") : secret "dns-default-metrics-tls" not found Sep 10 20:57:31.917466 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:31.917372 2574 generic.go:356] "Generic (PLEG): container finished" podID="544862f7-a4c6-4753-9932-22668843f699" containerID="3440905f2b071370879f0437b3d0df4da441f67e4bd7b3009680faf6f7cf5a0e" exitCode=0 Sep 10 20:57:31.917919 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:31.917456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerDied","Data":"3440905f2b071370879f0437b3d0df4da441f67e4bd7b3009680faf6f7cf5a0e"} Sep 10 20:57:31.918953 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:31.918925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rzdqm" event={"ID":"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d","Type":"ContainerStarted","Data":"a2ae0bb9e3e79c350b6f7d871e0df26db72ddc4ddfaba2a5ed540c31d8f36af8"} Sep 10 20:57:32.924064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:32.924034 2574 generic.go:356] "Generic (PLEG): container finished" podID="544862f7-a4c6-4753-9932-22668843f699" containerID="5d2447638a0704bd444759f8ac178ccb2b6a37bcc014de74ffc2ea4bd769eae4" exitCode=0 Sep 10 20:57:32.924490 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:32.924096 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerDied","Data":"5d2447638a0704bd444759f8ac178ccb2b6a37bcc014de74ffc2ea4bd769eae4"} Sep 10 20:57:33.240221 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:33.240133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:33.240365 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:33.240312 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 20:57:33.240365 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:33.240331 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b877979b8-6xrtf: secret "image-registry-tls" not found Sep 10 20:57:33.240459 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:33.240402 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls podName:5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:37.240380734 +0000 UTC m=+43.058954857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls") pod "image-registry-b877979b8-6xrtf" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58") : secret "image-registry-tls" not found Sep 10 20:57:33.341616 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:33.341577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:33.341791 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:33.341718 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:57:33.341791 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:33.341785 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls podName:597fc486-2ab7-49cb-91ae-9d4a37bd174b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:37.341767192 +0000 UTC m=+43.160341305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls") pod "dns-default-pbkpz" (UID: "597fc486-2ab7-49cb-91ae-9d4a37bd174b") : secret "dns-default-metrics-tls" not found Sep 10 20:57:33.930093 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:33.930063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-82h58" event={"ID":"544862f7-a4c6-4753-9932-22668843f699","Type":"ContainerStarted","Data":"29d3f4dd8e95e6b905d83d8a2112bde1ebe4641500d7ed7d8e58ff6137dc7796"} Sep 10 20:57:33.965191 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:33.965152 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-82h58" podStartSLOduration=8.450504235 podStartE2EDuration="38.965138478s" totalImagesPullingTime="30.514634243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:33.965079526 +0000 UTC m=+39.783653646" watchObservedRunningTime="2026-09-10 20:57:33.965138478 +0000 UTC m=+39.783712598" Sep 10 20:57:34.014493 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:34.014460 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:34.167986 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:34.167951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:34.303388 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:34.303352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:34.552633 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:34.552603 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:34.679176 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:34.679138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:34.934601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:34.934508 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rzdqm" event={"ID":"a91b9b2b-9609-4f4d-9bcd-07e184a8f19d","Type":"ContainerStarted","Data":"9469376b72562cd7b9a3ea437b07d11bc3d0f25c7b47063d9a1b18d4e851eaba"} Sep 10 20:57:34.956017 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:34.955962 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rzdqm" podStartSLOduration=3.377873329 podStartE2EDuration="5.955943652s" totalImagesPullingTime="2.578070323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:34.95499494 +0000 UTC m=+40.773569061" watchObservedRunningTime="2026-09-10 20:57:34.955943652 +0000 UTC m=+40.774517769" Sep 10 20:57:35.092575 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:35.092540 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:35.191135 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:35.191061 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:35.601285 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:35.601250 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:35.687592 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:35.687562 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:37.270892 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:37.270852 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:37.271249 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:37.271005 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 20:57:37.271249 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:37.271023 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b877979b8-6xrtf: secret "image-registry-tls" not found Sep 10 20:57:37.271249 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:37.271094 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls podName:5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.271077112 +0000 UTC m=+51.089651232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls") pod "image-registry-b877979b8-6xrtf" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58") : secret "image-registry-tls" not found Sep 10 20:57:37.322334 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:37.322300 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:37.371995 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:37.371959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:37.372132 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:37.372115 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:57:37.372199 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:37.372188 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls podName:597fc486-2ab7-49cb-91ae-9d4a37bd174b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.372169733 +0000 UTC m=+51.190743854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls") pod "dns-default-pbkpz" (UID: "597fc486-2ab7-49cb-91ae-9d4a37bd174b") : secret "dns-default-metrics-tls" not found Sep 10 20:57:37.444139 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:37.444106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:39.759459 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:39.759420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:39.885185 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:39.885147 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:41.606690 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.606658 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln"] Sep 10 20:57:41.627478 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.627451 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln"] Sep 10 20:57:41.627601 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.627551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.633753 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.633730 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 20:57:41.633905 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.633887 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 20:57:41.633973 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.633887 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 20:57:41.635794 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.635773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8qgkw\"" Sep 10 20:57:41.635914 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.635801 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 20:57:41.706927 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.706900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhkf\" (UniqueName: \"kubernetes.io/projected/fe947841-41c5-485b-8061-72984004c125-kube-api-access-pjhkf\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.707048 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.706949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fe947841-41c5-485b-8061-72984004c125-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.707048 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.706993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.807927 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.807897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhkf\" (UniqueName: \"kubernetes.io/projected/fe947841-41c5-485b-8061-72984004c125-kube-api-access-pjhkf\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.808046 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.807952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fe947841-41c5-485b-8061-72984004c125-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.808046 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.807986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.808114 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:41.808080 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:41.808157 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:41.808148 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls podName:fe947841-41c5-485b-8061-72984004c125 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:42.30813008 +0000 UTC m=+48.126704192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-djpln" (UID: "fe947841-41c5-485b-8061-72984004c125") : secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:41.808721 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.808703 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fe947841-41c5-485b-8061-72984004c125-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.843697 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.843673 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg"] Sep 10 20:57:41.861673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.861626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:41.867578 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.867557 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 20:57:41.867675 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.867631 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 20:57:41.867738 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.867726 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-txlrs\"" Sep 10 20:57:41.878051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.878032 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhkf\" (UniqueName: \"kubernetes.io/projected/fe947841-41c5-485b-8061-72984004c125-kube-api-access-pjhkf\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:41.886911 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:41.886893 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg"] Sep 10 20:57:42.010148 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.010100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:42.010148 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.010152 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18d9a069-cb74-40d1-ac8a-e774358f015a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:42.110687 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.110651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:42.110849 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.110694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18d9a069-cb74-40d1-ac8a-e774358f015a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:42.110849 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:42.110803 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:57:42.110942 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:42.110909 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert podName:18d9a069-cb74-40d1-ac8a-e774358f015a nodeName:}" failed. No retries permitted until 2026-09-10 20:57:42.610889423 +0000 UTC m=+48.429463536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l5wpg" (UID: "18d9a069-cb74-40d1-ac8a-e774358f015a") : secret "networking-console-plugin-cert" not found Sep 10 20:57:42.111431 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.111414 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18d9a069-cb74-40d1-ac8a-e774358f015a-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:42.195535 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.195434 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:42.312210 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.312176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:42.312348 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:42.312325 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:42.312410 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:42.312400 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls podName:fe947841-41c5-485b-8061-72984004c125 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:43.312384312 +0000 UTC m=+49.130958411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-djpln" (UID: "fe947841-41c5-485b-8061-72984004c125") : secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:42.324164 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.324138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:42.614587 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:42.614554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:42.614952 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:42.614679 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:57:42.614952 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:42.614738 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert podName:18d9a069-cb74-40d1-ac8a-e774358f015a nodeName:}" failed. No retries permitted until 2026-09-10 20:57:43.61472582 +0000 UTC m=+49.433299919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l5wpg" (UID: "18d9a069-cb74-40d1-ac8a-e774358f015a") : secret "networking-console-plugin-cert" not found Sep 10 20:57:43.319754 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:43.319710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:43.319958 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:43.319882 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:43.319958 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:43.319947 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls podName:fe947841-41c5-485b-8061-72984004c125 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.319932234 +0000 UTC m=+51.138506352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-djpln" (UID: "fe947841-41c5-485b-8061-72984004c125") : secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:43.621046 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:43.620969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:43.621370 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:43.621136 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:57:43.621370 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:43.621209 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert podName:18d9a069-cb74-40d1-ac8a-e774358f015a nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.621193171 +0000 UTC m=+51.439767293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l5wpg" (UID: "18d9a069-cb74-40d1-ac8a-e774358f015a") : secret "networking-console-plugin-cert" not found Sep 10 20:57:45.062263 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:45.062226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:45.189561 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:45.189528 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:45.336218 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:45.336138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:57:45.336218 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:45.336181 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:45.336416 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.336287 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 20:57:45.336416 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.336305 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b877979b8-6xrtf: secret "image-registry-tls" not found Sep 10 20:57:45.336416 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.336356 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:45.336416 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.336362 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls podName:5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:01.336345379 +0000 UTC m=+67.154919478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls") pod "image-registry-b877979b8-6xrtf" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58") : secret "image-registry-tls" not found Sep 10 20:57:45.336416 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.336414 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls podName:fe947841-41c5-485b-8061-72984004c125 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:49.336402528 +0000 UTC m=+55.154976627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-djpln" (UID: "fe947841-41c5-485b-8061-72984004c125") : secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:45.436924 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:45.436897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:57:45.437066 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.437043 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 20:57:45.437119 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.437108 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls podName:597fc486-2ab7-49cb-91ae-9d4a37bd174b nodeName:}" failed. No retries permitted until 2026-09-10 20:58:01.437094601 +0000 UTC m=+67.255668731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls") pod "dns-default-pbkpz" (UID: "597fc486-2ab7-49cb-91ae-9d4a37bd174b") : secret "dns-default-metrics-tls" not found Sep 10 20:57:45.639244 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:45.639160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:45.639405 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.639331 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:57:45.639466 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:45.639411 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert podName:18d9a069-cb74-40d1-ac8a-e774358f015a nodeName:}" failed. No retries permitted until 2026-09-10 20:57:49.639389756 +0000 UTC m=+55.457963868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l5wpg" (UID: "18d9a069-cb74-40d1-ac8a-e774358f015a") : secret "networking-console-plugin-cert" not found Sep 10 20:57:46.430122 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:46.430087 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:46.519842 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:46.519795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:46.645083 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:46.645050 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:46.763666 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:46.763603 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:47.144857 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:47.144826 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:47.249013 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:47.248979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:48.742803 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.742766 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-kndsk"] Sep 10 20:57:48.744662 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.744646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.748803 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.748778 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 20:57:48.749193 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.749175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 20:57:48.749756 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.749736 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 20:57:48.749900 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.749780 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 20:57:48.749900 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.749782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-994f5\"" Sep 10 20:57:48.764130 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.764110 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-kndsk"] Sep 10 20:57:48.818269 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.818242 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:48.862794 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.862768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26565d72-29c7-4ce3-a34d-d21b49a84baf-signing-cabundle\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.862925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.862802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26565d72-29c7-4ce3-a34d-d21b49a84baf-signing-key\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.862925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.862919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjs4v\" (UniqueName: \"kubernetes.io/projected/26565d72-29c7-4ce3-a34d-d21b49a84baf-kube-api-access-pjs4v\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.963319 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.963293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjs4v\" (UniqueName: \"kubernetes.io/projected/26565d72-29c7-4ce3-a34d-d21b49a84baf-kube-api-access-pjs4v\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.963423 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.963404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26565d72-29c7-4ce3-a34d-d21b49a84baf-signing-cabundle\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.963467 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.963453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26565d72-29c7-4ce3-a34d-d21b49a84baf-signing-key\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.964097 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.964066 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26565d72-29c7-4ce3-a34d-d21b49a84baf-signing-cabundle\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.966167 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.966148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26565d72-29c7-4ce3-a34d-d21b49a84baf-signing-key\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.975662 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.975642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjs4v\" (UniqueName: \"kubernetes.io/projected/26565d72-29c7-4ce3-a34d-d21b49a84baf-kube-api-access-pjs4v\") pod \"service-ca-568869fdc5-kndsk\" (UID: \"26565d72-29c7-4ce3-a34d-d21b49a84baf\") " pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:48.984115 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:48.984094 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:49.053209 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.053188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-kndsk" Sep 10 20:57:49.364162 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.364084 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-kndsk"] Sep 10 20:57:49.366533 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.366511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:49.366689 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.366668 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:49.366799 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.366787 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls podName:fe947841-41c5-485b-8061-72984004c125 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.366770049 +0000 UTC m=+63.185344150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-djpln" (UID: "fe947841-41c5-485b-8061-72984004c125") : secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:49.373001 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:57:49.372979 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26565d72_29c7_4ce3_a34d_d21b49a84baf.slice/crio-a436682a2d428f9b52ff0433f88c92f6d6cc03fa1c13e9e3fb15a33a87bba5d1 WatchSource:0}: Error finding container a436682a2d428f9b52ff0433f88c92f6d6cc03fa1c13e9e3fb15a33a87bba5d1: Status 404 returned error can't find the container with id a436682a2d428f9b52ff0433f88c92f6d6cc03fa1c13e9e3fb15a33a87bba5d1 Sep 10 20:57:49.478243 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.478218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:49.668791 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.668697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:49.668941 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.668853 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:57:49.668941 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.668914 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert podName:18d9a069-cb74-40d1-ac8a-e774358f015a nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.668899781 +0000 UTC m=+63.487473880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l5wpg" (UID: "18d9a069-cb74-40d1-ac8a-e774358f015a") : secret "networking-console-plugin-cert" not found Sep 10 20:57:49.729862 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.729828 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-289rz"] Sep 10 20:57:49.732658 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.732641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.734600 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.734578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:49.736998 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.736981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 20:57:49.737104 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.737067 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 20:57:49.738979 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.738965 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 20:57:49.740404 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.740382 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 20:57:49.744740 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.744723 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b4777\"" Sep 10 20:57:49.859593 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.859562 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-289rz"] Sep 10 20:57:49.869709 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.869687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-crio-socket\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.869874 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.869740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.869874 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.869791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64mh\" (UniqueName: \"kubernetes.io/projected/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-api-access-p64mh\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.869874 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.869861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-data-volume\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.870038 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.869879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.926128 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.926036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:49.969613 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.969574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-kndsk" event={"ID":"26565d72-29c7-4ce3-a34d-d21b49a84baf","Type":"ContainerStarted","Data":"a436682a2d428f9b52ff0433f88c92f6d6cc03fa1c13e9e3fb15a33a87bba5d1"} Sep 10 20:57:49.970823 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.970789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.970938 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.970842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64mh\" (UniqueName: \"kubernetes.io/projected/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-api-access-p64mh\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.970938 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.970894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-data-volume\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.970938 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.970917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.971097 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.970939 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:57:49.971097 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.970997 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:50.47097987 +0000 UTC m=+56.289553971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : secret "insights-runtime-extractor-tls" not found Sep 10 20:57:49.971097 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.971014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-crio-socket\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.971097 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.971021 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:57:49.971097 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:49.971072 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:50.471060492 +0000 UTC m=+56.289634609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : configmap "kube-rbac-proxy" not found Sep 10 20:57:49.971354 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.971110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-crio-socket\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.971405 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.971364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-data-volume\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:49.992536 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:49.992507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64mh\" (UniqueName: \"kubernetes.io/projected/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-api-access-p64mh\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:50.475051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:50.475016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:50.475226 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:50.475111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:50.475226 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:50.475205 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:57:50.475343 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:50.475288 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:51.475263367 +0000 UTC m=+57.293837488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : secret "insights-runtime-extractor-tls" not found Sep 10 20:57:50.475448 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:50.475205 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:57:50.475563 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:50.475500 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:51.475478709 +0000 UTC m=+57.294052833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : configmap "kube-rbac-proxy" not found Sep 10 20:57:50.618524 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:50.618483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:51.483236 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:51.483141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:51.483236 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:51.483214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:51.483738 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:51.483288 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:57:51.483738 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:51.483355 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:53.483339047 +0000 UTC m=+59.301913159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : configmap "kube-rbac-proxy" not found Sep 10 20:57:51.483738 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:51.483294 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:57:51.483738 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:51.483427 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:53.483413396 +0000 UTC m=+59.301987499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : secret "insights-runtime-extractor-tls" not found Sep 10 20:57:51.976088 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:51.976053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-kndsk" event={"ID":"26565d72-29c7-4ce3-a34d-d21b49a84baf","Type":"ContainerStarted","Data":"984a8826e3d62701579a650e2be680fcd750b2f478abcc842b9d2c1b1614dcbe"} Sep 10 20:57:51.999001 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:51.998953 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-kndsk" podStartSLOduration=2.217742991 podStartE2EDuration="3.998939118s" totalImagesPullingTime="1.781196127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:51.99333266 +0000 UTC m=+57.811906781" watchObservedRunningTime="2026-09-10 20:57:51.998939118 +0000 UTC m=+57.817513239" Sep 10 20:57:52.441713 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:52.441686 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tc7ff_b7b16656-f7d0-4f2e-9326-fa2dda31ff23/dns-node-resolver/0.log" Sep 10 20:57:52.738332 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:52.738232 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:52.972522 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:52.972486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:53.113642 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:53.113605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:53.373035 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:53.372945 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:53.475111 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:53.475079 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:53.503217 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:53.503186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:53.503357 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:53.503275 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:53.503357 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:53.503344 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:57:53.503463 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:53.503374 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:57:53.503463 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:53.503408 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.503391345 +0000 UTC m=+63.321965444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : secret "insights-runtime-extractor-tls" not found Sep 10 20:57:53.503463 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:53.503430 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.503414025 +0000 UTC m=+63.321988139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : configmap "kube-rbac-proxy" not found Sep 10 20:57:53.656223 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:53.656154 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pmgvp_b62fe6e0-e353-446e-af6f-89c408244412/node-ca/0.log" Sep 10 20:57:54.442241 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:54.442214 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rzdqm_a91b9b2b-9609-4f4d-9bcd-07e184a8f19d/serve-healthcheck-canary/0.log" Sep 10 20:57:54.663386 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:54.663349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:54.860647 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:54.860610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:55.060511 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:55.060478 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:55.231449 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:55.231362 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:55.477995 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:55.477962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:55.896937 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:55.896909 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:56.483646 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:56.483610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:56.637036 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:56.637007 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:56.869483 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:56.869449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:56.910138 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:56.910119 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xkwq7" Sep 10 20:57:57.089173 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.089138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:57.279477 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.279441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:57.437731 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.437685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:57:57.437941 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.437884 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:57.437994 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.437978 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls podName:fe947841-41c5-485b-8061-72984004c125 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:13.437956933 +0000 UTC m=+79.256531047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-djpln" (UID: "fe947841-41c5-485b-8061-72984004c125") : secret "cluster-monitoring-operator-tls" not found Sep 10 20:57:57.464580 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.464547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:57.538437 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.538341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:57.538934 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.538443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:57:57.538934 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.538480 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 20:57:57.538934 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.538542 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 20:57:57.538934 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.538583 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:05.538559805 +0000 UTC m=+71.357133924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : configmap "kube-rbac-proxy" not found Sep 10 20:57:57.538934 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.538606 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls podName:37516ba4-a1d3-41a7-b5b3-5e4ab1339957 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:05.53859555 +0000 UTC m=+71.357169662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289rz" (UID: "37516ba4-a1d3-41a7-b5b3-5e4ab1339957") : secret "insights-runtime-extractor-tls" not found Sep 10 20:57:57.658212 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.658181 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:57.739583 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:57.739543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:57:57.739755 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.739728 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 20:57:57.739857 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:57:57.739842 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert podName:18d9a069-cb74-40d1-ac8a-e774358f015a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:13.739794271 +0000 UTC m=+79.558368388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-l5wpg" (UID: "18d9a069-cb74-40d1-ac8a-e774358f015a") : secret "networking-console-plugin-cert" not found Sep 10 20:57:58.042836 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:58.042794 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:58.647083 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:58.647049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:59.069144 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:59.069107 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:59.202715 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:59.202678 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:59.448454 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:59.448348 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:57:59.681975 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:57:59.681942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:00.057613 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.057566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:00.360531 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.360439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:58:00.363079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.363055 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 20:58:00.374540 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.374501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cb1d07-086d-48a2-92a7-f8061493cead-metrics-certs\") pod \"network-metrics-daemon-kflkv\" (UID: \"d9cb1d07-086d-48a2-92a7-f8061493cead\") " pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:58:00.461003 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.460966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:58:00.464068 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.464046 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 20:58:00.474384 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.474354 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c4c92c2-36db-4308-b268-f01026044047-original-pull-secret\") pod \"global-pull-secret-syncer-jsfgg\" (UID: \"8c4c92c2-36db-4308-b268-f01026044047\") " pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:58:00.561343 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.561307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:58:00.566910 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.566889 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 20:58:00.575028 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.575001 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 20:58:00.585559 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.585529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4j5\" (UniqueName: \"kubernetes.io/projected/0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5-kube-api-access-2l4j5\") pod \"network-check-target-6rlj6\" (UID: \"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5\") " pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:58:00.629934 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.629828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:00.658998 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.658965 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zr4bv\"" Sep 10 20:58:00.662476 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.662454 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jsfgg" Sep 10 20:58:00.666091 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.666073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kflkv" Sep 10 20:58:00.858265 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.858234 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gncgs\"" Sep 10 20:58:00.865498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.865475 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:58:00.951642 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.951602 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kflkv"] Sep 10 20:58:00.956789 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:00.956759 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cb1d07_086d_48a2_92a7_f8061493cead.slice/crio-29be4f9ab1a1a199c4243eb4b45bcf37bf6d1b1fd2e33b0864f1be0ea0f1a7d9 WatchSource:0}: Error finding container 29be4f9ab1a1a199c4243eb4b45bcf37bf6d1b1fd2e33b0864f1be0ea0f1a7d9: Status 404 returned error can't find the container with id 29be4f9ab1a1a199c4243eb4b45bcf37bf6d1b1fd2e33b0864f1be0ea0f1a7d9 Sep 10 20:58:00.958104 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:00.958081 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jsfgg"] Sep 10 20:58:00.961256 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:00.961236 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4c92c2_36db_4308_b268_f01026044047.slice/crio-e0b47059066b3fcc2a1461bf2439742d2e1c955e7699e2f513b428491011d07d WatchSource:0}: Error finding container e0b47059066b3fcc2a1461bf2439742d2e1c955e7699e2f513b428491011d07d: Status 404 returned error can't find the container with id e0b47059066b3fcc2a1461bf2439742d2e1c955e7699e2f513b428491011d07d Sep 10 20:58:01.001452 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.001409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kflkv" event={"ID":"d9cb1d07-086d-48a2-92a7-f8061493cead","Type":"ContainerStarted","Data":"29be4f9ab1a1a199c4243eb4b45bcf37bf6d1b1fd2e33b0864f1be0ea0f1a7d9"} Sep 10 20:58:01.002603 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.002581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jsfgg" event={"ID":"8c4c92c2-36db-4308-b268-f01026044047","Type":"ContainerStarted","Data":"e0b47059066b3fcc2a1461bf2439742d2e1c955e7699e2f513b428491011d07d"} Sep 10 20:58:01.048955 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.048923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:01.141004 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.140980 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6rlj6"] Sep 10 20:58:01.142033 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:01.142002 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3ed048_38b2_4bdf_9f9f_7f9538e4dfa5.slice/crio-fa46db743f2f3a940eee0783d47b1b05327848e5b75f6eb22c5a00920debcb41 WatchSource:0}: Error finding container fa46db743f2f3a940eee0783d47b1b05327848e5b75f6eb22c5a00920debcb41: Status 404 returned error can't find the container with id fa46db743f2f3a940eee0783d47b1b05327848e5b75f6eb22c5a00920debcb41 Sep 10 20:58:01.223005 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.222909 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:01.368373 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.368330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:58:01.370902 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.370879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"image-registry-b877979b8-6xrtf\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:58:01.469719 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.469684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:58:01.473204 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.473144 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/597fc486-2ab7-49cb-91ae-9d4a37bd174b-metrics-tls\") pod \"dns-default-pbkpz\" (UID: \"597fc486-2ab7-49cb-91ae-9d4a37bd174b\") " pod="openshift-dns/dns-default-pbkpz" Sep 10 20:58:01.633477 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.633372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:01.648274 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.648240 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-24mm2\"" Sep 10 20:58:01.655759 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.655727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:58:01.707439 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.707408 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw26g\"" Sep 10 20:58:01.714337 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.714307 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbkpz" Sep 10 20:58:01.740902 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.740854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:01.970363 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:01.970323 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b877979b8-6xrtf"] Sep 10 20:58:02.007362 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:02.007278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6rlj6" event={"ID":"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5","Type":"ContainerStarted","Data":"fa46db743f2f3a940eee0783d47b1b05327848e5b75f6eb22c5a00920debcb41"} Sep 10 20:58:02.016497 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:02.016459 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbkpz"] Sep 10 20:58:02.102576 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:02.102537 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5951ba84_64e9_4e3d_a4a5_f8b6f0cbda58.slice/crio-ba3d83b43ee852b3500c38526c0cfeec3d5531182ee63d4da61dd8c8291aead1 WatchSource:0}: Error finding container ba3d83b43ee852b3500c38526c0cfeec3d5531182ee63d4da61dd8c8291aead1: Status 404 returned error can't find the container with id ba3d83b43ee852b3500c38526c0cfeec3d5531182ee63d4da61dd8c8291aead1 Sep 10 20:58:02.104022 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:02.103990 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597fc486_2ab7_49cb_91ae_9d4a37bd174b.slice/crio-9d3f67876e9345cae0292c0b99e1d3a265df55c003212a6669593bc8f2769ae5 WatchSource:0}: Error finding container 9d3f67876e9345cae0292c0b99e1d3a265df55c003212a6669593bc8f2769ae5: Status 404 returned error can't find the container with id 9d3f67876e9345cae0292c0b99e1d3a265df55c003212a6669593bc8f2769ae5 Sep 10 20:58:02.447428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:02.447390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:02.847455 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:02.847423 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:03.012849 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.012771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbkpz" event={"ID":"597fc486-2ab7-49cb-91ae-9d4a37bd174b","Type":"ContainerStarted","Data":"9d3f67876e9345cae0292c0b99e1d3a265df55c003212a6669593bc8f2769ae5"} Sep 10 20:58:03.015361 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.015329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" event={"ID":"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58","Type":"ContainerStarted","Data":"0edaa907a51e2066db9836f6746fd67a149b9016068e5a849ae1f86382785e71"} Sep 10 20:58:03.015549 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.015389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" event={"ID":"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58","Type":"ContainerStarted","Data":"ba3d83b43ee852b3500c38526c0cfeec3d5531182ee63d4da61dd8c8291aead1"} Sep 10 20:58:03.015549 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.015420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:58:03.018756 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.018677 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kflkv" event={"ID":"d9cb1d07-086d-48a2-92a7-f8061493cead","Type":"ContainerStarted","Data":"5a1c59a8cea388f8c581ab37f38186d2f9aaf0d7e304c52c308b184feab451a8"} Sep 10 20:58:03.018756 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.018723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kflkv" event={"ID":"d9cb1d07-086d-48a2-92a7-f8061493cead","Type":"ContainerStarted","Data":"5c2ae4026f2f0d5ba3b16bec336c03268505e2f7aaef9bbc903b46dbbd23edb5"} Sep 10 20:58:03.056494 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.056459 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:03.062231 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.062146 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" podStartSLOduration=67.062124832 podStartE2EDuration="1m7.062124832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:03.061485461 +0000 UTC m=+68.880059582" watchObservedRunningTime="2026-09-10 20:58:03.062124832 +0000 UTC m=+68.880698955" Sep 10 20:58:03.096426 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.096366 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kflkv" podStartSLOduration=66.917304548 podStartE2EDuration="1m8.096346984s" totalImagesPullingTime="1.179042436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:03.096232218 +0000 UTC m=+68.914806417" watchObservedRunningTime="2026-09-10 20:58:03.096346984 +0000 UTC m=+68.914921106" Sep 10 20:58:03.280644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.280613 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:03.404703 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.404594 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:03.508830 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:03.508706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:05.610668 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:05.610631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:58:05.611134 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:05.610731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:58:05.611264 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:05.611234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:58:05.613458 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:05.613415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37516ba4-a1d3-41a7-b5b3-5e4ab1339957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289rz\" (UID: \"37516ba4-a1d3-41a7-b5b3-5e4ab1339957\") " pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:58:05.650561 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:05.650530 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b4777\"" Sep 10 20:58:05.651572 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:05.651545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-289rz" Sep 10 20:58:07.032087 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:07.032048 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6rlj6" event={"ID":"0f3ed048-38b2-4bdf-9f9f-7f9538e4dfa5","Type":"ContainerStarted","Data":"71a40ce40e3a2d3ea78a0addf331d8d41874f9a7cc2613f2b30ddad89642263a"} Sep 10 20:58:07.032605 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:07.032562 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:58:07.034424 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:07.034400 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jsfgg" event={"ID":"8c4c92c2-36db-4308-b268-f01026044047","Type":"ContainerStarted","Data":"6392d193013e7c385609ea4a47f4a97cc5deb84720c0ef2889f3adfa6e79cb73"} Sep 10 20:58:07.042424 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:07.042401 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbkpz" event={"ID":"597fc486-2ab7-49cb-91ae-9d4a37bd174b","Type":"ContainerStarted","Data":"5d9ad4d6e8f30f484ef6a35afcf0d1087f30ca97360404082ac43a2b07cd38a9"} Sep 10 20:58:07.068260 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:07.068208 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6rlj6" podStartSLOduration=66.378664215 podStartE2EDuration="1m12.068189967s" totalImagesPullingTime="5.689525752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:07.066787873 +0000 UTC m=+72.885361993" watchObservedRunningTime="2026-09-10 20:58:07.068189967 +0000 UTC m=+72.886764090" Sep 10 20:58:07.098552 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:07.098500 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jsfgg" podStartSLOduration=65.214019235 podStartE2EDuration="1m11.098479436s" totalImagesPullingTime="5.884460201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:07.097193052 +0000 UTC m=+72.915767172" watchObservedRunningTime="2026-09-10 20:58:07.098479436 +0000 UTC m=+72.917053557" Sep 10 20:58:07.119303 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:07.119263 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-289rz"] Sep 10 20:58:07.122884 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:07.122856 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37516ba4_a1d3_41a7_b5b3_5e4ab1339957.slice/crio-ed35a3953be042b7a1afda977edbc14446216957ce831c87b209080ec236a8df WatchSource:0}: Error finding container ed35a3953be042b7a1afda977edbc14446216957ce831c87b209080ec236a8df: Status 404 returned error can't find the container with id ed35a3953be042b7a1afda977edbc14446216957ce831c87b209080ec236a8df Sep 10 20:58:08.047083 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:08.047050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbkpz" event={"ID":"597fc486-2ab7-49cb-91ae-9d4a37bd174b","Type":"ContainerStarted","Data":"a00a0be0314ab09da3c3c7901b953208b78602f4252161b6afd6208c48b82c92"} Sep 10 20:58:08.047472 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:08.047146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pbkpz" Sep 10 20:58:08.048956 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:08.048936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289rz" event={"ID":"37516ba4-a1d3-41a7-b5b3-5e4ab1339957","Type":"ContainerStarted","Data":"7b70c8a890abc74e87d1063ca8b87e7b72700da93b1fb4ba8aa32febd342f557"} Sep 10 20:58:08.049043 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:08.048985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289rz" event={"ID":"37516ba4-a1d3-41a7-b5b3-5e4ab1339957","Type":"ContainerStarted","Data":"1a20eca4940771f8a60589ed42f4d095bd0c4e08a770a436fd069ae8619ba258"} Sep 10 20:58:08.049043 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:08.049004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289rz" event={"ID":"37516ba4-a1d3-41a7-b5b3-5e4ab1339957","Type":"ContainerStarted","Data":"ed35a3953be042b7a1afda977edbc14446216957ce831c87b209080ec236a8df"} Sep 10 20:58:08.075691 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:08.075643 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pbkpz" podStartSLOduration=34.349886566 podStartE2EDuration="39.075629251s" totalImagesPullingTime="4.725742685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:08.075039226 +0000 UTC m=+73.893613348" watchObservedRunningTime="2026-09-10 20:58:08.075629251 +0000 UTC m=+73.894203372" Sep 10 20:58:09.898689 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:09.898658 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:10.058341 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:10.058302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289rz" event={"ID":"37516ba4-a1d3-41a7-b5b3-5e4ab1339957","Type":"ContainerStarted","Data":"6f22db4aa4ef8ee3ad7d9220f6323b583641a85504cc8f0333d5f18979835aa3"} Sep 10 20:58:10.075747 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:10.075720 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:13.466037 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.465980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:58:13.468426 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.468403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe947841-41c5-485b-8061-72984004c125-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-djpln\" (UID: \"fe947841-41c5-485b-8061-72984004c125\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:58:13.740776 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.740695 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8qgkw\"" Sep 10 20:58:13.748183 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.748165 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" Sep 10 20:58:13.768375 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.768353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:58:13.770851 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.770781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18d9a069-cb74-40d1-ac8a-e774358f015a-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-l5wpg\" (UID: \"18d9a069-cb74-40d1-ac8a-e774358f015a\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:58:13.972865 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.972840 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-txlrs\"" Sep 10 20:58:13.980440 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:13.980421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" Sep 10 20:58:14.039926 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.039879 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-289rz" podStartSLOduration=23.342065036 podStartE2EDuration="25.039860336s" totalImagesPullingTime="1.6977953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:10.086556402 +0000 UTC m=+75.905130522" watchObservedRunningTime="2026-09-10 20:58:14.039860336 +0000 UTC m=+79.858434458" Sep 10 20:58:14.040414 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.040395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln"] Sep 10 20:58:14.043247 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:14.043220 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe947841_41c5_485b_8061_72984004c125.slice/crio-2088d86cd90b644bf18cde7f05d88a73a6f6c47d6e1872f8dbf154ee964fa1fb WatchSource:0}: Error finding container 2088d86cd90b644bf18cde7f05d88a73a6f6c47d6e1872f8dbf154ee964fa1fb: Status 404 returned error can't find the container with id 2088d86cd90b644bf18cde7f05d88a73a6f6c47d6e1872f8dbf154ee964fa1fb Sep 10 20:58:14.072141 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.072114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" event={"ID":"fe947841-41c5-485b-8061-72984004c125","Type":"ContainerStarted","Data":"2088d86cd90b644bf18cde7f05d88a73a6f6c47d6e1872f8dbf154ee964fa1fb"} Sep 10 20:58:14.293630 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.293607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg"] Sep 10 20:58:14.294995 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:14.294963 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d9a069_cb74_40d1_ac8a_e774358f015a.slice/crio-fa352e0d0246c5f11ecc241b0b6bf183f7f60a3821eb0d7d521d35f05864b4b3 WatchSource:0}: Error finding container fa352e0d0246c5f11ecc241b0b6bf183f7f60a3821eb0d7d521d35f05864b4b3: Status 404 returned error can't find the container with id fa352e0d0246c5f11ecc241b0b6bf183f7f60a3821eb0d7d521d35f05864b4b3 Sep 10 20:58:14.917409 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.917372 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4"] Sep 10 20:58:14.920067 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.919450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:14.925062 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.925038 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 20:58:14.926286 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.926039 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 20:58:14.926286 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.926063 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 20:58:14.926286 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.926122 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 20:58:14.956975 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.956948 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4"] Sep 10 20:58:14.977595 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.977439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-tmp\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:14.977595 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.977566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p9sw\" (UniqueName: \"kubernetes.io/projected/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-kube-api-access-9p9sw\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:14.977790 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:14.977648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-klusterlet-config\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.077113 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.077080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" event={"ID":"18d9a069-cb74-40d1-ac8a-e774358f015a","Type":"ContainerStarted","Data":"fa352e0d0246c5f11ecc241b0b6bf183f7f60a3821eb0d7d521d35f05864b4b3"} Sep 10 20:58:15.078709 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.078676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-klusterlet-config\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.078864 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.078737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-tmp\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.078864 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.078850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p9sw\" (UniqueName: \"kubernetes.io/projected/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-kube-api-access-9p9sw\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.079585 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.079563 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-tmp\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.085841 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.085794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-klusterlet-config\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.104618 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.104590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:15.105842 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.105797 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p9sw\" (UniqueName: \"kubernetes.io/projected/3a8737cd-d9fd-4e06-ac53-878a7b30b2bf-kube-api-access-9p9sw\") pod \"klusterlet-addon-workmgr-79cfc46d44-5bvp4\" (UID: \"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.232624 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.232330 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:15.268091 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.268004 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:15.561375 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.561326 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:15.756818 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:15.756779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:16.082829 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.082779 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" event={"ID":"18d9a069-cb74-40d1-ac8a-e774358f015a","Type":"ContainerStarted","Data":"4e86c0076bf5b7e985c1740d6e0c721b0f3bfaa23c562edabf26501239396c80"} Sep 10 20:58:16.084586 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.084562 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" event={"ID":"fe947841-41c5-485b-8061-72984004c125","Type":"ContainerStarted","Data":"9326a18f353dc9db21c94ce3e8bdf438e96b4dcba5a997df1e632d57c5a11aaa"} Sep 10 20:58:16.122497 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.122430 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-l5wpg" podStartSLOduration=33.442177157 podStartE2EDuration="35.122413899s" totalImagesPullingTime="1.680236742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:16.121205049 +0000 UTC m=+81.939779171" watchObservedRunningTime="2026-09-10 20:58:16.122413899 +0000 UTC m=+81.940988017" Sep 10 20:58:16.271954 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.271898 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-djpln" podStartSLOduration=33.336108842 podStartE2EDuration="35.271877022s" totalImagesPullingTime="1.93576818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:16.155383719 +0000 UTC m=+81.973957842" watchObservedRunningTime="2026-09-10 20:58:16.271877022 +0000 UTC m=+82.090451144" Sep 10 20:58:16.272648 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.272626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4"] Sep 10 20:58:16.272764 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:16.272744 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8737cd_d9fd_4e06_ac53_878a7b30b2bf.slice/crio-94b8d69c411de5459d967be5440644cb717c30a314672f2bded87116695dc06d WatchSource:0}: Error finding container 94b8d69c411de5459d967be5440644cb717c30a314672f2bded87116695dc06d: Status 404 returned error can't find the container with id 94b8d69c411de5459d967be5440644cb717c30a314672f2bded87116695dc06d Sep 10 20:58:16.412317 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.412285 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:16.636366 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.636332 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:16.766311 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.766270 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:16.928483 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.928417 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj"] Sep 10 20:58:16.930968 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.930953 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" Sep 10 20:58:16.937674 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.937326 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 20:58:16.937674 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.937498 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jd2pw\"" Sep 10 20:58:16.963879 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.963853 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj"] Sep 10 20:58:16.990668 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:16.990638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/655c6939-cce7-4971-ae37-3cd37d2b628f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pnqrj\" (UID: \"655c6939-cce7-4971-ae37-3cd37d2b628f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" Sep 10 20:58:17.013068 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:17.013045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:17.090673 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:17.090641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" event={"ID":"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf","Type":"ContainerStarted","Data":"94b8d69c411de5459d967be5440644cb717c30a314672f2bded87116695dc06d"} Sep 10 20:58:17.091147 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:17.091117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/655c6939-cce7-4971-ae37-3cd37d2b628f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pnqrj\" (UID: \"655c6939-cce7-4971-ae37-3cd37d2b628f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" Sep 10 20:58:17.093912 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:17.093886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/655c6939-cce7-4971-ae37-3cd37d2b628f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pnqrj\" (UID: \"655c6939-cce7-4971-ae37-3cd37d2b628f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" Sep 10 20:58:17.173362 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:17.173308 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:17.240706 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:17.240417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" Sep 10 20:58:17.539106 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:17.539014 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj"] Sep 10 20:58:17.542177 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:17.542145 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655c6939_cce7_4971_ae37_3cd37d2b628f.slice/crio-9b78fb0d02cddc897d20bcdf68fcf46cd89f6d9eba81b751542ae94ac054bd7f WatchSource:0}: Error finding container 9b78fb0d02cddc897d20bcdf68fcf46cd89f6d9eba81b751542ae94ac054bd7f: Status 404 returned error can't find the container with id 9b78fb0d02cddc897d20bcdf68fcf46cd89f6d9eba81b751542ae94ac054bd7f Sep 10 20:58:18.054619 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:18.054593 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pbkpz" Sep 10 20:58:18.096184 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:18.096151 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" event={"ID":"655c6939-cce7-4971-ae37-3cd37d2b628f","Type":"ContainerStarted","Data":"9b78fb0d02cddc897d20bcdf68fcf46cd89f6d9eba81b751542ae94ac054bd7f"} Sep 10 20:58:21.112088 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.112046 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" event={"ID":"655c6939-cce7-4971-ae37-3cd37d2b628f","Type":"ContainerStarted","Data":"40b535095bf08c2555d3dbc289751dcfa126c3d766cb1892c4ceb96afc847a39"} Sep 10 20:58:21.112550 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.112239 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" Sep 10 20:58:21.113822 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.113774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" event={"ID":"3a8737cd-d9fd-4e06-ac53-878a7b30b2bf","Type":"ContainerStarted","Data":"84fe205721c395490bf698056466b21a6049cedbb5805e9a3b266ebfd6e1e37b"} Sep 10 20:58:21.114078 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.114045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:21.115369 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.115351 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" Sep 10 20:58:21.117071 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.117055 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" Sep 10 20:58:21.141042 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.141003 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pnqrj" podStartSLOduration=2.409317955 podStartE2EDuration="5.1409898s" totalImagesPullingTime="2.731671845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:21.140043531 +0000 UTC m=+86.958617652" watchObservedRunningTime="2026-09-10 20:58:21.1409898 +0000 UTC m=+86.959563917" Sep 10 20:58:21.177264 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.177219 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cfc46d44-5bvp4" podStartSLOduration=3.16257352 podStartE2EDuration="7.177206858s" totalImagesPullingTime="4.014633338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:21.177136757 +0000 UTC m=+86.995710878" watchObservedRunningTime="2026-09-10 20:58:21.177206858 +0000 UTC m=+86.995780980" Sep 10 20:58:21.660064 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.660023 2574 patch_prober.go:28] interesting pod/image-registry-b877979b8-6xrtf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 20:58:21.660224 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.660083 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" podUID="5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 20:58:21.945644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:21.945564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:22.054778 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.054740 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-84qt2"] Sep 10 20:58:22.057378 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.057357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.060853 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.060833 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 20:58:22.061801 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.061777 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 20:58:22.061934 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.061782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-498vn\"" Sep 10 20:58:22.061934 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.061802 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 20:58:22.075786 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.075766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-84qt2"] Sep 10 20:58:22.104035 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.104008 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:22.129578 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.129553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqkc\" (UniqueName: \"kubernetes.io/projected/3f6ca02d-da18-4fa9-b783-7b0bbed28928-kube-api-access-tnqkc\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.129905 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.129581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6ca02d-da18-4fa9-b783-7b0bbed28928-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.129905 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.129602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f6ca02d-da18-4fa9-b783-7b0bbed28928-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.129905 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.129619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ca02d-da18-4fa9-b783-7b0bbed28928-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.230117 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.230041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqkc\" (UniqueName: \"kubernetes.io/projected/3f6ca02d-da18-4fa9-b783-7b0bbed28928-kube-api-access-tnqkc\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.230117 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.230086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6ca02d-da18-4fa9-b783-7b0bbed28928-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.230297 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.230276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f6ca02d-da18-4fa9-b783-7b0bbed28928-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.230337 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.230311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ca02d-da18-4fa9-b783-7b0bbed28928-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.230984 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.230961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6ca02d-da18-4fa9-b783-7b0bbed28928-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.232579 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.232550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6ca02d-da18-4fa9-b783-7b0bbed28928-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.232676 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.232640 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3f6ca02d-da18-4fa9-b783-7b0bbed28928-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.242066 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.242046 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqkc\" (UniqueName: \"kubernetes.io/projected/3f6ca02d-da18-4fa9-b783-7b0bbed28928-kube-api-access-tnqkc\") pod \"prometheus-operator-68bcd7f987-84qt2\" (UID: \"3f6ca02d-da18-4fa9-b783-7b0bbed28928\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.248259 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.248233 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:22.365989 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.365962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" Sep 10 20:58:22.641125 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:22.641096 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-84qt2"] Sep 10 20:58:22.644121 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:22.644097 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6ca02d_da18_4fa9_b783_7b0bbed28928.slice/crio-165bfb2056dfe2f2fca4c6ac9c1fef6adf7af3252bdcc25e816380a8e8ea9a9e WatchSource:0}: Error finding container 165bfb2056dfe2f2fca4c6ac9c1fef6adf7af3252bdcc25e816380a8e8ea9a9e: Status 404 returned error can't find the container with id 165bfb2056dfe2f2fca4c6ac9c1fef6adf7af3252bdcc25e816380a8e8ea9a9e Sep 10 20:58:23.121008 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:23.120970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" event={"ID":"3f6ca02d-da18-4fa9-b783-7b0bbed28928","Type":"ContainerStarted","Data":"165bfb2056dfe2f2fca4c6ac9c1fef6adf7af3252bdcc25e816380a8e8ea9a9e"} Sep 10 20:58:23.240311 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:23.240273 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:23.361565 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:23.361537 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:24.026661 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:24.026629 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:58:24.303591 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:24.303549 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:24.432866 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:24.432832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:24.539060 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:24.538974 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:25.130196 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:25.130161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" event={"ID":"3f6ca02d-da18-4fa9-b783-7b0bbed28928","Type":"ContainerStarted","Data":"c875733b0b55317f101d15d4d7b0ac9458a9cf1ae60c308ce955e6734572ca66"} Sep 10 20:58:25.130360 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:25.130215 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" event={"ID":"3f6ca02d-da18-4fa9-b783-7b0bbed28928","Type":"ContainerStarted","Data":"76091d17e199cbfab0d8c00cc38919509e5b295faf5334d644ab0c41ee70ed52"} Sep 10 20:58:25.156575 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:25.156525 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-84qt2" podStartSLOduration=1.5960914609999999 podStartE2EDuration="3.156499423s" totalImagesPullingTime="1.560407962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:25.155544468 +0000 UTC m=+90.974118590" watchObservedRunningTime="2026-09-10 20:58:25.156499423 +0000 UTC m=+90.975073544" Sep 10 20:58:26.570980 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:26.570937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:26.848761 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:26.848661 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:27.380937 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:27.380900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:27.846716 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:27.846681 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:28.049787 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.049754 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv"] Sep 10 20:58:28.053291 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.053259 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j8cbq"] Sep 10 20:58:28.053438 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.053408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.054629 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.054600 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:28.056652 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.056630 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 20:58:28.056779 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.056660 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ffqh5\"" Sep 10 20:58:28.057032 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.057011 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.057609 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.057594 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 20:58:28.060056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.060037 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 20:58:28.060199 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.060178 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 20:58:28.060306 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.060268 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tgnsq\"" Sep 10 20:58:28.064052 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.064033 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv"] Sep 10 20:58:28.066171 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.066152 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 20:58:28.074884 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.074866 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 20:58:28.174324 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a74ba8c-089a-40a5-8321-da76e624fe3d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.174324 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174292 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68f4\" (UniqueName: \"kubernetes.io/projected/37091e43-3957-4338-90a8-8c51e7f709ee-kube-api-access-n68f4\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.174324 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174318 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.174589 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.174589 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-tls\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.174589 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.174589 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37091e43-3957-4338-90a8-8c51e7f709ee-metrics-client-ca\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.174589 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5bm\" (UniqueName: \"kubernetes.io/projected/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-api-access-bh5bm\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.174873 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-sys\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.174873 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-wtmp\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.174873 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.174873 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-textfile\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.175077 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a74ba8c-089a-40a5-8321-da76e624fe3d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.175077 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-root\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.175077 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.174955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276218 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.276218 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-textfile\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276409 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a74ba8c-089a-40a5-8321-da76e624fe3d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.276409 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-root\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a74ba8c-089a-40a5-8321-da76e624fe3d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.276564 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68f4\" (UniqueName: \"kubernetes.io/projected/37091e43-3957-4338-90a8-8c51e7f709ee-kube-api-access-n68f4\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276564 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276500 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-textfile\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276564 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276508 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-root\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276564 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.276743 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276743 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-tls\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276743 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.276743 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37091e43-3957-4338-90a8-8c51e7f709ee-metrics-client-ca\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276743 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276688 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5bm\" (UniqueName: \"kubernetes.io/projected/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-api-access-bh5bm\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.276743 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-sys\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.276743 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-wtmp\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.277105 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.276862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-wtmp\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.277105 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.277089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.277211 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.277114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37091e43-3957-4338-90a8-8c51e7f709ee-sys\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.277211 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.277135 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.277368 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.277343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2a74ba8c-089a-40a5-8321-da76e624fe3d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.277464 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.277440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37091e43-3957-4338-90a8-8c51e7f709ee-metrics-client-ca\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.277914 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.277877 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a74ba8c-089a-40a5-8321-da76e624fe3d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.279084 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.279062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.279216 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.279199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.279460 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.279443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.279890 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.279874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37091e43-3957-4338-90a8-8c51e7f709ee-node-exporter-tls\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.297333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.297315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5bm\" (UniqueName: \"kubernetes.io/projected/2a74ba8c-089a-40a5-8321-da76e624fe3d-kube-api-access-bh5bm\") pod \"kube-state-metrics-59fbdd77d-m7djv\" (UID: \"2a74ba8c-089a-40a5-8321-da76e624fe3d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.311696 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.311669 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68f4\" (UniqueName: \"kubernetes.io/projected/37091e43-3957-4338-90a8-8c51e7f709ee-kube-api-access-n68f4\") pod \"node-exporter-j8cbq\" (UID: \"37091e43-3957-4338-90a8-8c51e7f709ee\") " pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.364803 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.364779 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" Sep 10 20:58:28.371388 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.371369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j8cbq" Sep 10 20:58:28.382788 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:28.382762 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37091e43_3957_4338_90a8_8c51e7f709ee.slice/crio-bd045d0ce3095969ca07522244a9fba3fea4dffcceded47b0f7ab523c5864fc1 WatchSource:0}: Error finding container bd045d0ce3095969ca07522244a9fba3fea4dffcceded47b0f7ab523c5864fc1: Status 404 returned error can't find the container with id bd045d0ce3095969ca07522244a9fba3fea4dffcceded47b0f7ab523c5864fc1 Sep 10 20:58:28.401161 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.401128 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:28.669867 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.669843 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv"] Sep 10 20:58:28.672230 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:28.672202 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a74ba8c_089a_40a5_8321_da76e624fe3d.slice/crio-db69954e817dc4b8f43780d50824e80a02ea0e553fbc21cd92d8b1c962a67916 WatchSource:0}: Error finding container db69954e817dc4b8f43780d50824e80a02ea0e553fbc21cd92d8b1c962a67916: Status 404 returned error can't find the container with id db69954e817dc4b8f43780d50824e80a02ea0e553fbc21cd92d8b1c962a67916 Sep 10 20:58:28.786384 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:28.786336 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:29.009826 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:29.009721 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:29.150396 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:29.150347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" event={"ID":"2a74ba8c-089a-40a5-8321-da76e624fe3d","Type":"ContainerStarted","Data":"db69954e817dc4b8f43780d50824e80a02ea0e553fbc21cd92d8b1c962a67916"} Sep 10 20:58:29.151953 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:29.151888 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j8cbq" event={"ID":"37091e43-3957-4338-90a8-8c51e7f709ee","Type":"ContainerStarted","Data":"bd045d0ce3095969ca07522244a9fba3fea4dffcceded47b0f7ab523c5864fc1"} Sep 10 20:58:29.199322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:29.199288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:29.625696 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:29.625659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:29.963835 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:29.963762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:30.157998 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:30.157967 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" event={"ID":"2a74ba8c-089a-40a5-8321-da76e624fe3d","Type":"ContainerStarted","Data":"4a167c596f67e81aff87ab8a720c3555d6a2d64cd897871ff7d52eefa404c158"} Sep 10 20:58:30.160010 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:30.159968 2574 generic.go:356] "Generic (PLEG): container finished" podID="37091e43-3957-4338-90a8-8c51e7f709ee" containerID="28d01004c06ef33e7b7c160f76d4ce970691c4f263efc943f7a4268788057d69" exitCode=0 Sep 10 20:58:30.160109 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:30.160017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j8cbq" event={"ID":"37091e43-3957-4338-90a8-8c51e7f709ee","Type":"ContainerDied","Data":"28d01004c06ef33e7b7c160f76d4ce970691c4f263efc943f7a4268788057d69"} Sep 10 20:58:31.166377 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.166334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" event={"ID":"2a74ba8c-089a-40a5-8321-da76e624fe3d","Type":"ContainerStarted","Data":"dfc250545d327155acce560a6464eaff2bbd5b6334caf2e3cf0b5ed549001aa9"} Sep 10 20:58:31.166851 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.166401 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" event={"ID":"2a74ba8c-089a-40a5-8321-da76e624fe3d","Type":"ContainerStarted","Data":"d6ac0851f3c3e91f6a159f3197b8be31e767df82af00727586cd0e86f1377751"} Sep 10 20:58:31.168509 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.168482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j8cbq" event={"ID":"37091e43-3957-4338-90a8-8c51e7f709ee","Type":"ContainerStarted","Data":"9ca98a62ae0ea02aabab300e93a46f173e5a0eaead43ceedf06022041ecc92c0"} Sep 10 20:58:31.168634 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.168531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j8cbq" event={"ID":"37091e43-3957-4338-90a8-8c51e7f709ee","Type":"ContainerStarted","Data":"84c304512703119c27c6c264290052fee173fc0b27d594741f982e740be23c45"} Sep 10 20:58:31.190045 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.190007 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-m7djv" podStartSLOduration=2.775959993 podStartE2EDuration="4.189993357s" totalImagesPullingTime="1.414033364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:31.188703318 +0000 UTC m=+97.007277440" watchObservedRunningTime="2026-09-10 20:58:31.189993357 +0000 UTC m=+97.008567478" Sep 10 20:58:31.227116 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.227073 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j8cbq" podStartSLOduration=3.041643993 podStartE2EDuration="4.227056734s" totalImagesPullingTime="1.109277849s" totalInitContainerRuntime="76.134892ms" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:31.226660405 +0000 UTC m=+97.045234527" watchObservedRunningTime="2026-09-10 20:58:31.227056734 +0000 UTC m=+97.045630854" Sep 10 20:58:31.464319 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.464241 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:31.708445 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.708407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:31.913957 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:31.913918 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:32.119149 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.119113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:32.207324 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.207249 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d5976855f-ktlsq"] Sep 10 20:58:32.210752 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.210736 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.214079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.214053 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 20:58:32.214226 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.214213 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aa5jcubfemvbu\"" Sep 10 20:58:32.214555 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.214539 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 20:58:32.216492 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.216473 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m6g8t\"" Sep 10 20:58:32.216706 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.216692 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 20:58:32.216778 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.216716 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 20:58:32.218374 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.218353 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 20:58:32.265625 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.265597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d5976855f-ktlsq"] Sep 10 20:58:32.311389 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.311358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-tls\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.311627 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.311604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.311683 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.311667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.311752 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.311717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-grpc-tls\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.311915 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.311841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e80058f1-e403-4771-b1ff-0d0206ba1f1a-metrics-client-ca\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.311915 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.311902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.312028 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.311936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.312071 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.312049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjfg\" (UniqueName: \"kubernetes.io/projected/e80058f1-e403-4771-b1ff-0d0206ba1f1a-kube-api-access-svjfg\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.389668 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.389631 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:32.412407 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.412407 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.412585 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-grpc-tls\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.412585 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e80058f1-e403-4771-b1ff-0d0206ba1f1a-metrics-client-ca\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.412585 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.412585 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.412585 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjfg\" (UniqueName: \"kubernetes.io/projected/e80058f1-e403-4771-b1ff-0d0206ba1f1a-kube-api-access-svjfg\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.412843 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.412665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-tls\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.413656 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.413591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e80058f1-e403-4771-b1ff-0d0206ba1f1a-metrics-client-ca\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.415545 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.415332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.415545 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.415447 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-tls\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.415705 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.415566 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-grpc-tls\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.415991 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.415970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.416176 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.416155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.416729 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.416712 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e80058f1-e403-4771-b1ff-0d0206ba1f1a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.430301 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.430274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjfg\" (UniqueName: \"kubernetes.io/projected/e80058f1-e403-4771-b1ff-0d0206ba1f1a-kube-api-access-svjfg\") pod \"thanos-querier-5d5976855f-ktlsq\" (UID: \"e80058f1-e403-4771-b1ff-0d0206ba1f1a\") " pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.521309 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.521234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:32.840181 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.840147 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d5976855f-ktlsq"] Sep 10 20:58:32.844059 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:32.844035 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80058f1_e403_4771_b1ff_0d0206ba1f1a.slice/crio-d3d77504a0d7ee5b22d4ca260444a5dd0d1c09bcec1ca976bbd97cbc4857d88f WatchSource:0}: Error finding container d3d77504a0d7ee5b22d4ca260444a5dd0d1c09bcec1ca976bbd97cbc4857d88f: Status 404 returned error can't find the container with id d3d77504a0d7ee5b22d4ca260444a5dd0d1c09bcec1ca976bbd97cbc4857d88f Sep 10 20:58:32.866752 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:32.866721 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:33.045282 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.045254 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b858df774-xzbxq"] Sep 10 20:58:33.049801 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.049780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.052427 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:33.052400 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-dockercfg-2lw95\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2lw95\"" type="*v1.Secret" Sep 10 20:58:33.052535 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:33.052477 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-3e07v08drios0\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-server-3e07v08drios0\"" type="*v1.Secret" Sep 10 20:58:33.052663 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:33.052641 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-tls\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" Sep 10 20:58:33.052927 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:33.052909 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-client-certs\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" Sep 10 20:58:33.053548 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:33.053519 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"metrics-server-5b858df774-xzbxq\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" podUID="a00d1f1b-56de-47df-8413-26a7d4e9cf05" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.053917 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.053894 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 20:58:33.063310 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:33.063287 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kubelet-serving-ca-bundle\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" Sep 10 20:58:33.073146 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.073125 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b858df774-xzbxq"] Sep 10 20:58:33.118074 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.118014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a00d1f1b-56de-47df-8413-26a7d4e9cf05-audit-log\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.118074 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.118044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a00d1f1b-56de-47df-8413-26a7d4e9cf05-metrics-server-audit-profiles\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.118257 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.118080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-client-certs\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.118257 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.118110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00d1f1b-56de-47df-8413-26a7d4e9cf05-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.118257 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.118132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-client-ca-bundle\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.118257 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.118161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvkv\" (UniqueName: \"kubernetes.io/projected/a00d1f1b-56de-47df-8413-26a7d4e9cf05-kube-api-access-8hvkv\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.118257 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.118211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-tls\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.177649 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.177616 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" event={"ID":"e80058f1-e403-4771-b1ff-0d0206ba1f1a","Type":"ContainerStarted","Data":"d3d77504a0d7ee5b22d4ca260444a5dd0d1c09bcec1ca976bbd97cbc4857d88f"} Sep 10 20:58:33.219083 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a00d1f1b-56de-47df-8413-26a7d4e9cf05-audit-log\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.219428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a00d1f1b-56de-47df-8413-26a7d4e9cf05-metrics-server-audit-profiles\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.219428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-client-certs\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.219428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00d1f1b-56de-47df-8413-26a7d4e9cf05-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.219428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-client-ca-bundle\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.219428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvkv\" (UniqueName: \"kubernetes.io/projected/a00d1f1b-56de-47df-8413-26a7d4e9cf05-kube-api-access-8hvkv\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.219428 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-tls\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.219629 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.219450 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a00d1f1b-56de-47df-8413-26a7d4e9cf05-audit-log\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.220184 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.220166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a00d1f1b-56de-47df-8413-26a7d4e9cf05-metrics-server-audit-profiles\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.229315 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.229292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvkv\" (UniqueName: \"kubernetes.io/projected/a00d1f1b-56de-47df-8413-26a7d4e9cf05-kube-api-access-8hvkv\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:33.920433 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.920403 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3e07v08drios0\"" Sep 10 20:58:33.932538 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:33.932514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-client-ca-bundle\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:34.046909 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.046882 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 20:58:34.050476 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.050452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00d1f1b-56de-47df-8413-26a7d4e9cf05-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:34.220259 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:34.220177 2574 secret.go:189] Couldn't get secret openshift-monitoring/metrics-server-client-certs: failed to sync secret cache: timed out waiting for the condition Sep 10 20:58:34.220259 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:34.220202 2574 secret.go:189] Couldn't get secret openshift-monitoring/metrics-server-tls: failed to sync secret cache: timed out waiting for the condition Sep 10 20:58:34.220751 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:34.220268 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-client-certs podName:a00d1f1b-56de-47df-8413-26a7d4e9cf05 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:34.720249237 +0000 UTC m=+100.538823337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-metrics-server-client-certs" (UniqueName: "kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-client-certs") pod "metrics-server-5b858df774-xzbxq" (UID: "a00d1f1b-56de-47df-8413-26a7d4e9cf05") : failed to sync secret cache: timed out waiting for the condition Sep 10 20:58:34.220751 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:58:34.220282 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-tls podName:a00d1f1b-56de-47df-8413-26a7d4e9cf05 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:34.720276028 +0000 UTC m=+100.538850127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-metrics-server-tls" (UniqueName: "kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-tls") pod "metrics-server-5b858df774-xzbxq" (UID: "a00d1f1b-56de-47df-8413-26a7d4e9cf05") : failed to sync secret cache: timed out waiting for the condition Sep 10 20:58:34.509848 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.509755 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2lw95\"" Sep 10 20:58:34.559592 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.559556 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 20:58:34.654096 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.654068 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 20:58:34.733536 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.733494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-client-certs\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:34.733751 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.733572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-tls\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:34.736432 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.736409 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-tls\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:34.736543 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.736500 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a00d1f1b-56de-47df-8413-26a7d4e9cf05-secret-metrics-server-client-certs\") pod \"metrics-server-5b858df774-xzbxq\" (UID: \"a00d1f1b-56de-47df-8413-26a7d4e9cf05\") " pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:34.857300 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.857129 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:58:34.859245 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.859220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:34.863800 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.863421 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:34.872397 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.872371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:34.877303 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.877272 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 20:58:34.877410 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.877364 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6fn2g\"" Sep 10 20:58:34.877410 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.877382 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 20:58:34.878288 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.878271 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 20:58:34.878365 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.878271 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 20:58:34.878540 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.878528 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ci5vn4gq6ku2k\"" Sep 10 20:58:34.878686 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.878673 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 20:58:34.878852 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.878835 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 20:58:34.878975 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.878961 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 20:58:34.879028 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.878999 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 20:58:34.879291 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.879272 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 20:58:34.879913 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.879897 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 20:58:34.881059 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.881041 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 20:58:34.883776 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.883562 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 20:58:34.890772 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.886740 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 20:58:34.892068 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.892043 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:58:34.892418 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:34.892385 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 20:58:35.039681 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.039643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.039963 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.039934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52pr\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-kube-api-access-k52pr\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.039990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040098 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.040598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.040461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.141831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.141831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.141831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.141831 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.141987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142922 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142922 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52pr\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-kube-api-access-k52pr\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.142922 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.142896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.144110 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.143106 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.144110 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.143516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.144110 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.143769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.145304 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.145278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.145879 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.145794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.145982 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.145916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config-out\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.146320 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.146288 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.147185 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.147139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.147864 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.147786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.148342 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.148320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.148486 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.148443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.148486 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.148344 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.149671 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.149637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.150494 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.150009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.150494 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.150482 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.151054 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.150849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.151217 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.151193 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-web-config\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.152278 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.152258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.153331 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.153308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52pr\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-kube-api-access-k52pr\") pod \"prometheus-k8s-0\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.183737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.183712 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:35.208691 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.208661 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b858df774-xzbxq"] Sep 10 20:58:35.212074 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.212047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:35.281631 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:35.281599 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00d1f1b_56de_47df_8413_26a7d4e9cf05.slice/crio-40c4c07086e9eb10eb3db017a089fc10f18a82124455bcdf28b13584988a886d WatchSource:0}: Error finding container 40c4c07086e9eb10eb3db017a089fc10f18a82124455bcdf28b13584988a886d: Status 404 returned error can't find the container with id 40c4c07086e9eb10eb3db017a089fc10f18a82124455bcdf28b13584988a886d Sep 10 20:58:35.414916 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.414848 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:35.559923 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.559886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:35.597723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:35.597628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:58:35.602216 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:58:35.602195 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0b1e8d_c240_4d16_ad26_7e09d88b7da9.slice/crio-4a98c39e9b9c528ebe3843e06994274f0f6864846fca1c9e3de51a17a7fe2d26 WatchSource:0}: Error finding container 4a98c39e9b9c528ebe3843e06994274f0f6864846fca1c9e3de51a17a7fe2d26: Status 404 returned error can't find the container with id 4a98c39e9b9c528ebe3843e06994274f0f6864846fca1c9e3de51a17a7fe2d26 Sep 10 20:58:36.192399 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.192316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" event={"ID":"a00d1f1b-56de-47df-8413-26a7d4e9cf05","Type":"ContainerStarted","Data":"40c4c07086e9eb10eb3db017a089fc10f18a82124455bcdf28b13584988a886d"} Sep 10 20:58:36.194386 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.194287 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerStarted","Data":"4a98c39e9b9c528ebe3843e06994274f0f6864846fca1c9e3de51a17a7fe2d26"} Sep 10 20:58:36.197569 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.197543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" event={"ID":"e80058f1-e403-4771-b1ff-0d0206ba1f1a","Type":"ContainerStarted","Data":"33b7eb8de291e6c7dc5187d58f8dedf38935b89c62a414b40f39aaac7e67b8f7"} Sep 10 20:58:36.197692 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.197609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" event={"ID":"e80058f1-e403-4771-b1ff-0d0206ba1f1a","Type":"ContainerStarted","Data":"4c2ae5f69a1ebf3fe0464585d2842ce722df3a40ab348ccb3c2102c18b001b06"} Sep 10 20:58:36.197692 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.197653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" event={"ID":"e80058f1-e403-4771-b1ff-0d0206ba1f1a","Type":"ContainerStarted","Data":"19e7c31194289aefc984e020033764806bfc95c180148ecfd91a0ee99250d1ab"} Sep 10 20:58:36.477267 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.477113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:36.591582 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.591542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:36.751929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:36.751700 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:37.039718 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:37.039681 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:37.171328 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:37.171290 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b877979b8-6xrtf"] Sep 10 20:58:37.261110 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:37.261077 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:37.410139 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:37.410052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:37.644600 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:37.644564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:37.955401 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:37.955377 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:38.096051 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.096023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:38.208056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.208023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" event={"ID":"a00d1f1b-56de-47df-8413-26a7d4e9cf05","Type":"ContainerStarted","Data":"776f9284a0b57b419e815f411c531d8f507fe4ed3504784401084f9c85c0c2da"} Sep 10 20:58:38.209606 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.209584 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" exitCode=0 Sep 10 20:58:38.209704 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.209661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995"} Sep 10 20:58:38.212532 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.212512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" event={"ID":"e80058f1-e403-4771-b1ff-0d0206ba1f1a","Type":"ContainerStarted","Data":"b73b4a6489ec1dfe0d607fe69ce48a3c12a118d6481e6224d71b2a710f8e8c29"} Sep 10 20:58:38.212627 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.212576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" event={"ID":"e80058f1-e403-4771-b1ff-0d0206ba1f1a","Type":"ContainerStarted","Data":"fc073e14093572a07565a445d6465c753f82d8e55cc483ac33f3af8474bdfdd5"} Sep 10 20:58:38.238080 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.238039 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" podStartSLOduration=2.60834624 podStartE2EDuration="5.238024717s" totalImagesPullingTime="2.629678477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:38.235783136 +0000 UTC m=+104.054357257" watchObservedRunningTime="2026-09-10 20:58:38.238024717 +0000 UTC m=+104.056598904" Sep 10 20:58:38.247678 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:38.247611 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:39.055130 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:39.055093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6rlj6" Sep 10 20:58:39.221451 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:39.221371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" event={"ID":"e80058f1-e403-4771-b1ff-0d0206ba1f1a","Type":"ContainerStarted","Data":"a6163e59923fee2a8b9d3e4ecce696224439a4a3d52ae829262e4c99396851ca"} Sep 10 20:58:39.419888 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:39.419839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:39.583424 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:39.583389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:39.837757 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:39.837638 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:39.986705 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:39.986669 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:40.225887 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:40.225795 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:40.233379 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:40.233355 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" Sep 10 20:58:40.289036 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:40.288842 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d5976855f-ktlsq" podStartSLOduration=3.578878307 podStartE2EDuration="8.288820705s" totalImagesPullingTime="4.709942398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:39.265597515 +0000 UTC m=+105.084171637" watchObservedRunningTime="2026-09-10 20:58:40.288820705 +0000 UTC m=+106.107394819" Sep 10 20:58:42.237328 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:42.237290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerStarted","Data":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} Sep 10 20:58:42.237737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:42.237395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerStarted","Data":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} Sep 10 20:58:42.237737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:42.237424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerStarted","Data":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} Sep 10 20:58:42.237737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:42.237447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerStarted","Data":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} Sep 10 20:58:42.237737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:42.237474 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerStarted","Data":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} Sep 10 20:58:42.237737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:42.237498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerStarted","Data":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} Sep 10 20:58:42.297195 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:42.297147 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.699303667 podStartE2EDuration="8.297131152s" totalImagesPullingTime="5.552413669s" totalInitContainerRuntime="45.413816ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:42.295993821 +0000 UTC m=+108.114567968" watchObservedRunningTime="2026-09-10 20:58:42.297131152 +0000 UTC m=+108.115705272" Sep 10 20:58:45.184238 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:45.184184 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:58:45.266556 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:45.266519 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:45.379953 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:45.379921 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:51.016589 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:51.016544 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:51.108580 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:51.108546 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:51.232127 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:51.232088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:51.379410 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:51.379379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:53.822737 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:53.822709 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:53.943845 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:53.943793 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:54.859313 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:54.859280 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:54.859646 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:54.859337 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:58:58.070706 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:58.070667 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:58:58.255953 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:58:58.255922 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:02.192716 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.192675 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" podUID="5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" containerName="registry" containerID="cri-o://0edaa907a51e2066db9836f6746fd67a149b9016068e5a849ae1f86382785e71" gracePeriod=30 Sep 10 20:59:02.316457 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.316431 2574 generic.go:356] "Generic (PLEG): container finished" podID="5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" containerID="0edaa907a51e2066db9836f6746fd67a149b9016068e5a849ae1f86382785e71" exitCode=0 Sep 10 20:59:02.316593 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.316515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" event={"ID":"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58","Type":"ContainerDied","Data":"0edaa907a51e2066db9836f6746fd67a149b9016068e5a849ae1f86382785e71"} Sep 10 20:59:02.449724 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.449651 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:59:02.470351 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470305 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkjjd\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-kube-api-access-kkjjd\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470351 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-trusted-ca\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470382 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470506 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-image-registry-private-configuration\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470598 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470591 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-bound-sa-token\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470759 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470618 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-ca-trust-extracted\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470759 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470659 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-certificates\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470759 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470718 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-installation-pull-secrets\") pod \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\" (UID: \"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58\") " Sep 10 20:59:02.470931 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.470865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:02.471322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.471277 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-trusted-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:02.471456 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.471428 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:02.473191 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.473163 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:02.473498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.473450 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 20:59:02.473626 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.473555 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 20:59:02.473751 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.473729 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:02.473858 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.473836 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-kube-api-access-kkjjd" (OuterVolumeSpecName: "kube-api-access-kkjjd") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "kube-api-access-kkjjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 20:59:02.479299 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.479276 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" (UID: "5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 20:59:02.571967 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.571934 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-bound-sa-token\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:02.571967 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.571959 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-ca-trust-extracted\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:02.571967 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.571968 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-certificates\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:02.571967 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.571977 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-installation-pull-secrets\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:02.572222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.572008 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjjd\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-kube-api-access-kkjjd\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:02.572222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.572018 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-registry-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:02.572222 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:02.572026 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58-image-registry-private-configuration\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:03.321426 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:03.321399 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" Sep 10 20:59:03.321426 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:03.321412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b877979b8-6xrtf" event={"ID":"5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58","Type":"ContainerDied","Data":"ba3d83b43ee852b3500c38526c0cfeec3d5531182ee63d4da61dd8c8291aead1"} Sep 10 20:59:03.321944 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:03.321463 2574 scope.go:117] "RemoveContainer" containerID="0edaa907a51e2066db9836f6746fd67a149b9016068e5a849ae1f86382785e71" Sep 10 20:59:03.368049 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:03.368025 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b877979b8-6xrtf"] Sep 10 20:59:03.375378 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:03.375344 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b877979b8-6xrtf"] Sep 10 20:59:03.754546 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:03.754466 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:03.862318 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:03.862286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:04.649653 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:04.649624 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" path="/var/lib/kubelet/pods/5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58/volumes" Sep 10 20:59:09.771955 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:09.771917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:09.887154 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:09.887121 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:10.054364 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:10.054317 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rzdqm_a91b9b2b-9609-4f4d-9bcd-07e184a8f19d/serve-healthcheck-canary/0.log" Sep 10 20:59:14.864517 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:14.864488 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:59:14.868174 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:14.868156 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b858df774-xzbxq" Sep 10 20:59:18.740928 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:18.740894 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:18.883502 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:18.883473 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:18.981469 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:18.981441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:35.183987 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:35.183952 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:35.203667 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:35.203642 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:35.466297 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:35.466228 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:48.089867 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:48.089766 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:49.047331 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:49.047294 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:53.758135 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:53.758096 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:59:53.758727 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:53.758401 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="prometheus" containerID="cri-o://713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" gracePeriod=600 Sep 10 20:59:53.758727 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:53.758427 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="thanos-sidecar" containerID="cri-o://2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" gracePeriod=600 Sep 10 20:59:53.758727 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:53.758428 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy" containerID="cri-o://bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" gracePeriod=600 Sep 10 20:59:53.758727 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:53.758476 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="config-reloader" containerID="cri-o://76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" gracePeriod=600 Sep 10 20:59:53.758727 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:53.758499 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" gracePeriod=600 Sep 10 20:59:53.758727 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:53.758428 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-web" containerID="cri-o://29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" gracePeriod=600 Sep 10 20:59:54.009700 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.009642 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.119672 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.119633 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config-out\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.119869 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.119683 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-db\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.119869 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.119721 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.119869 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.119759 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-metrics-client-certs\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.119869 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.119791 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-thanos-prometheus-http-client-file\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120084 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.119892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-kubelet-serving-ca-bundle\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120084 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.119977 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-trusted-ca-bundle\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120084 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120008 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-web-config\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120084 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120037 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-0\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120084 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120060 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-tls\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120114 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-metrics-client-ca\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120146 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-grpc-tls\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120181 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120209 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-2\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120234 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-serving-certs-ca-bundle\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120290 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-kube-rbac-proxy\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120333 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120313 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-tls-assets\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120707 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120361 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-1\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120707 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120359 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:54.120707 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120391 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k52pr\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-kube-api-access-k52pr\") pod \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\" (UID: \"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9\") " Sep 10 20:59:54.120707 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120651 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:54.120926 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120873 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.120926 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120894 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.121027 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.120949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 20:59:54.121320 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.121288 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:54.121729 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.121699 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:54.123106 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.123064 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:54.123246 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.123218 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:54.123423 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.123384 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.123498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.123454 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.123758 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.123732 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-kube-api-access-k52pr" (OuterVolumeSpecName: "kube-api-access-k52pr") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "kube-api-access-k52pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 20:59:54.124077 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.124048 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.124283 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.124253 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config" (OuterVolumeSpecName: "config") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.124524 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.124480 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.124635 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.124609 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config-out" (OuterVolumeSpecName: "config-out") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 20:59:54.124929 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.124850 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 20:59:54.125041 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.124924 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.126248 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.126224 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 20:59:54.126356 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.126253 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.127120 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.127095 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.137278 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.137255 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-web-config" (OuterVolumeSpecName: "web-config") pod "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" (UID: "8a0b1e8d-c240-4d16-ad26-7e09d88b7da9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 20:59:54.221644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221608 2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-metrics-client-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221642 2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221657 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-web-config\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221671 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221686 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221697 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-metrics-client-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221708 2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-grpc-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221719 2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221729 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221742 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221755 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221767 2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-kube-rbac-proxy\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221778 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-tls-assets\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221790 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221831 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k52pr\" (UniqueName: \"kubernetes.io/projected/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-kube-api-access-k52pr\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221847 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-config-out\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221858 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-prometheus-k8s-db\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.221925 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.221870 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 20:59:54.524195 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524161 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" exitCode=0 Sep 10 20:59:54.524195 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524189 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" exitCode=0 Sep 10 20:59:54.524195 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524200 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" exitCode=0 Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524208 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" exitCode=0 Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524215 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" exitCode=0 Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524222 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" exitCode=0 Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524254 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} Sep 10 20:59:54.524479 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524475 2574 scope.go:117] "RemoveContainer" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.524910 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} Sep 10 20:59:54.524910 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} Sep 10 20:59:54.524910 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} Sep 10 20:59:54.524910 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.524618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8a0b1e8d-c240-4d16-ad26-7e09d88b7da9","Type":"ContainerDied","Data":"4a98c39e9b9c528ebe3843e06994274f0f6864846fca1c9e3de51a17a7fe2d26"} Sep 10 20:59:54.532768 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.532750 2574 scope.go:117] "RemoveContainer" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.539656 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.539640 2574 scope.go:117] "RemoveContainer" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.556555 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.556531 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:59:54.557878 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.557852 2574 scope.go:117] "RemoveContainer" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.565663 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.565644 2574 scope.go:117] "RemoveContainer" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.566644 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.566627 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:59:54.572122 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.572104 2574 scope.go:117] "RemoveContainer" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.578790 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.578776 2574 scope.go:117] "RemoveContainer" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.585248 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.585230 2574 scope.go:117] "RemoveContainer" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.585519 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:59:54.585499 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": container with ID starting with 10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc not found: ID does not exist" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.585595 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.585531 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} err="failed to get container status \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": rpc error: code = NotFound desc = could not find container \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": container with ID starting with 10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc not found: ID does not exist" Sep 10 20:59:54.585595 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.585581 2574 scope.go:117] "RemoveContainer" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.585886 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:59:54.585864 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": container with ID starting with bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36 not found: ID does not exist" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.585945 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.585895 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} err="failed to get container status \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": rpc error: code = NotFound desc = could not find container \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": container with ID starting with bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36 not found: ID does not exist" Sep 10 20:59:54.585945 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.585916 2574 scope.go:117] "RemoveContainer" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.586190 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:59:54.586163 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": container with ID starting with 29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18 not found: ID does not exist" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.586232 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.586196 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} err="failed to get container status \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": rpc error: code = NotFound desc = could not find container \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": container with ID starting with 29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18 not found: ID does not exist" Sep 10 20:59:54.586232 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.586210 2574 scope.go:117] "RemoveContainer" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.586443 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:59:54.586427 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": container with ID starting with 2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f not found: ID does not exist" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.586507 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.586450 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} err="failed to get container status \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": rpc error: code = NotFound desc = could not find container \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": container with ID starting with 2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f not found: ID does not exist" Sep 10 20:59:54.586507 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.586472 2574 scope.go:117] "RemoveContainer" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.586725 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:59:54.586708 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": container with ID starting with 76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110 not found: ID does not exist" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.586785 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.586731 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} err="failed to get container status \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": rpc error: code = NotFound desc = could not find container \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": container with ID starting with 76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110 not found: ID does not exist" Sep 10 20:59:54.586785 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.586748 2574 scope.go:117] "RemoveContainer" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.587026 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:59:54.587012 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": container with ID starting with 713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f not found: ID does not exist" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.587067 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587038 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} err="failed to get container status \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": rpc error: code = NotFound desc = could not find container \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": container with ID starting with 713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f not found: ID does not exist" Sep 10 20:59:54.587067 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587052 2574 scope.go:117] "RemoveContainer" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.587263 ip-10-0-132-255 kubenswrapper[2574]: E0910 20:59:54.587245 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": container with ID starting with 7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995 not found: ID does not exist" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.587300 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587268 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995"} err="failed to get container status \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": rpc error: code = NotFound desc = could not find container \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": container with ID starting with 7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995 not found: ID does not exist" Sep 10 20:59:54.587300 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587281 2574 scope.go:117] "RemoveContainer" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.587490 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587475 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} err="failed to get container status \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": rpc error: code = NotFound desc = could not find container \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": container with ID starting with 10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc not found: ID does not exist" Sep 10 20:59:54.587531 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587490 2574 scope.go:117] "RemoveContainer" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.587706 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587685 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} err="failed to get container status \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": rpc error: code = NotFound desc = could not find container \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": container with ID starting with bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36 not found: ID does not exist" Sep 10 20:59:54.587774 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587708 2574 scope.go:117] "RemoveContainer" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.587949 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587931 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} err="failed to get container status \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": rpc error: code = NotFound desc = could not find container \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": container with ID starting with 29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18 not found: ID does not exist" Sep 10 20:59:54.588002 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.587950 2574 scope.go:117] "RemoveContainer" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.588172 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588156 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} err="failed to get container status \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": rpc error: code = NotFound desc = could not find container \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": container with ID starting with 2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f not found: ID does not exist" Sep 10 20:59:54.588215 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588172 2574 scope.go:117] "RemoveContainer" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.588375 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588358 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} err="failed to get container status \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": rpc error: code = NotFound desc = could not find container \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": container with ID starting with 76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110 not found: ID does not exist" Sep 10 20:59:54.588443 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588376 2574 scope.go:117] "RemoveContainer" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.588627 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588600 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} err="failed to get container status \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": rpc error: code = NotFound desc = could not find container \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": container with ID starting with 713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f not found: ID does not exist" Sep 10 20:59:54.588627 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588618 2574 scope.go:117] "RemoveContainer" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.588829 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588796 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995"} err="failed to get container status \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": rpc error: code = NotFound desc = could not find container \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": container with ID starting with 7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995 not found: ID does not exist" Sep 10 20:59:54.588892 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.588831 2574 scope.go:117] "RemoveContainer" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.589076 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589059 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} err="failed to get container status \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": rpc error: code = NotFound desc = could not find container \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": container with ID starting with 10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc not found: ID does not exist" Sep 10 20:59:54.589120 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589078 2574 scope.go:117] "RemoveContainer" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.589277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589260 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} err="failed to get container status \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": rpc error: code = NotFound desc = could not find container \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": container with ID starting with bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36 not found: ID does not exist" Sep 10 20:59:54.589321 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589277 2574 scope.go:117] "RemoveContainer" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.589491 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589474 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} err="failed to get container status \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": rpc error: code = NotFound desc = could not find container \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": container with ID starting with 29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18 not found: ID does not exist" Sep 10 20:59:54.589565 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589493 2574 scope.go:117] "RemoveContainer" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.589746 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589730 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} err="failed to get container status \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": rpc error: code = NotFound desc = could not find container \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": container with ID starting with 2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f not found: ID does not exist" Sep 10 20:59:54.589793 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589746 2574 scope.go:117] "RemoveContainer" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.589989 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589971 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} err="failed to get container status \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": rpc error: code = NotFound desc = could not find container \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": container with ID starting with 76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110 not found: ID does not exist" Sep 10 20:59:54.590034 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.589990 2574 scope.go:117] "RemoveContainer" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.590213 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590196 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} err="failed to get container status \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": rpc error: code = NotFound desc = could not find container \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": container with ID starting with 713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f not found: ID does not exist" Sep 10 20:59:54.590277 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590215 2574 scope.go:117] "RemoveContainer" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.590450 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590434 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995"} err="failed to get container status \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": rpc error: code = NotFound desc = could not find container \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": container with ID starting with 7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995 not found: ID does not exist" Sep 10 20:59:54.590494 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590450 2574 scope.go:117] "RemoveContainer" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.590646 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590627 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} err="failed to get container status \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": rpc error: code = NotFound desc = could not find container \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": container with ID starting with 10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc not found: ID does not exist" Sep 10 20:59:54.590689 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590648 2574 scope.go:117] "RemoveContainer" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.590848 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590831 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} err="failed to get container status \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": rpc error: code = NotFound desc = could not find container \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": container with ID starting with bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36 not found: ID does not exist" Sep 10 20:59:54.590899 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.590848 2574 scope.go:117] "RemoveContainer" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.591067 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591050 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} err="failed to get container status \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": rpc error: code = NotFound desc = could not find container \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": container with ID starting with 29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18 not found: ID does not exist" Sep 10 20:59:54.591135 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591068 2574 scope.go:117] "RemoveContainer" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.591306 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591288 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} err="failed to get container status \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": rpc error: code = NotFound desc = could not find container \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": container with ID starting with 2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f not found: ID does not exist" Sep 10 20:59:54.591364 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591307 2574 scope.go:117] "RemoveContainer" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.591511 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591495 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} err="failed to get container status \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": rpc error: code = NotFound desc = could not find container \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": container with ID starting with 76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110 not found: ID does not exist" Sep 10 20:59:54.591553 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591511 2574 scope.go:117] "RemoveContainer" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.591723 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591706 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} err="failed to get container status \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": rpc error: code = NotFound desc = could not find container \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": container with ID starting with 713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f not found: ID does not exist" Sep 10 20:59:54.591768 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591724 2574 scope.go:117] "RemoveContainer" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.591951 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591935 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995"} err="failed to get container status \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": rpc error: code = NotFound desc = could not find container \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": container with ID starting with 7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995 not found: ID does not exist" Sep 10 20:59:54.592003 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.591952 2574 scope.go:117] "RemoveContainer" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.592125 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592110 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} err="failed to get container status \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": rpc error: code = NotFound desc = could not find container \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": container with ID starting with 10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc not found: ID does not exist" Sep 10 20:59:54.592165 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592125 2574 scope.go:117] "RemoveContainer" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.592350 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592331 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} err="failed to get container status \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": rpc error: code = NotFound desc = could not find container \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": container with ID starting with bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36 not found: ID does not exist" Sep 10 20:59:54.592395 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592351 2574 scope.go:117] "RemoveContainer" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.592589 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592571 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} err="failed to get container status \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": rpc error: code = NotFound desc = could not find container \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": container with ID starting with 29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18 not found: ID does not exist" Sep 10 20:59:54.592629 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592590 2574 scope.go:117] "RemoveContainer" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.592824 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592788 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} err="failed to get container status \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": rpc error: code = NotFound desc = could not find container \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": container with ID starting with 2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f not found: ID does not exist" Sep 10 20:59:54.592874 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.592824 2574 scope.go:117] "RemoveContainer" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.593066 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593046 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} err="failed to get container status \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": rpc error: code = NotFound desc = could not find container \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": container with ID starting with 76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110 not found: ID does not exist" Sep 10 20:59:54.593107 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593066 2574 scope.go:117] "RemoveContainer" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.593304 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593288 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} err="failed to get container status \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": rpc error: code = NotFound desc = could not find container \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": container with ID starting with 713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f not found: ID does not exist" Sep 10 20:59:54.593358 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593304 2574 scope.go:117] "RemoveContainer" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.593509 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593493 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995"} err="failed to get container status \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": rpc error: code = NotFound desc = could not find container \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": container with ID starting with 7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995 not found: ID does not exist" Sep 10 20:59:54.593560 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593510 2574 scope.go:117] "RemoveContainer" containerID="10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc" Sep 10 20:59:54.593733 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593717 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc"} err="failed to get container status \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": rpc error: code = NotFound desc = could not find container \"10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc\": container with ID starting with 10bd51f99d75e13f35da6af44c92c4977d78625b44b864a4c6a6b93056c243bc not found: ID does not exist" Sep 10 20:59:54.593775 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.593734 2574 scope.go:117] "RemoveContainer" containerID="bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36" Sep 10 20:59:54.594018 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594003 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36"} err="failed to get container status \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": rpc error: code = NotFound desc = could not find container \"bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36\": container with ID starting with bea0846e14aefee577bce6e33b636e1218fb5c33f58d514795efa4f09e39ac36 not found: ID does not exist" Sep 10 20:59:54.594075 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594018 2574 scope.go:117] "RemoveContainer" containerID="29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18" Sep 10 20:59:54.594244 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594228 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18"} err="failed to get container status \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": rpc error: code = NotFound desc = could not find container \"29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18\": container with ID starting with 29391886b958d37c61ad6e5d07fb5ee7621bb928aa93563202e59514ddb4fc18 not found: ID does not exist" Sep 10 20:59:54.594294 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594243 2574 scope.go:117] "RemoveContainer" containerID="2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f" Sep 10 20:59:54.594467 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594448 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f"} err="failed to get container status \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": rpc error: code = NotFound desc = could not find container \"2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f\": container with ID starting with 2a8b2004a1b0910a7daad577cc31487da9b100d603582277443a8464c33f506f not found: ID does not exist" Sep 10 20:59:54.594505 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594468 2574 scope.go:117] "RemoveContainer" containerID="76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110" Sep 10 20:59:54.594691 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594672 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110"} err="failed to get container status \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": rpc error: code = NotFound desc = could not find container \"76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110\": container with ID starting with 76ed542fe677e9dd896e0062923f7815fa12b4bb771ce2227a267f7f3fd0e110 not found: ID does not exist" Sep 10 20:59:54.594691 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594690 2574 scope.go:117] "RemoveContainer" containerID="713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f" Sep 10 20:59:54.594933 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594916 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f"} err="failed to get container status \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": rpc error: code = NotFound desc = could not find container \"713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f\": container with ID starting with 713a6d24de8674c08e4111d9d22b417fdab7109ac65cfd832a36093a5b2ebd1f not found: ID does not exist" Sep 10 20:59:54.594985 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.594934 2574 scope.go:117] "RemoveContainer" containerID="7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995" Sep 10 20:59:54.595157 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.595140 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995"} err="failed to get container status \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": rpc error: code = NotFound desc = could not find container \"7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995\": container with ID starting with 7934eeefab4d8ab48e49e0ba236c1b2c989b34a503ac8f10b1152b88ebbf4995 not found: ID does not exist" Sep 10 20:59:54.622492 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622468 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:59:54.622845 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622829 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" containerName="registry" Sep 10 20:59:54.622845 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622847 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" containerName="registry" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622864 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="init-config-reloader" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622870 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="init-config-reloader" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622879 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="config-reloader" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622885 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="config-reloader" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622893 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-web" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622898 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-web" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622905 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622910 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622919 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="thanos-sidecar" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622924 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="thanos-sidecar" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622929 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-thanos" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622933 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-thanos" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622939 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="prometheus" Sep 10 20:59:54.622940 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622944 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="prometheus" Sep 10 20:59:54.623322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622990 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="prometheus" Sep 10 20:59:54.623322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.622998 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5951ba84-64e9-4e3d-a4a5-f8b6f0cbda58" containerName="registry" Sep 10 20:59:54.623322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.623004 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="config-reloader" Sep 10 20:59:54.623322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.623010 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="thanos-sidecar" Sep 10 20:59:54.623322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.623014 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-thanos" Sep 10 20:59:54.623322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.623019 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy-web" Sep 10 20:59:54.623322 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.623024 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" containerName="kube-rbac-proxy" Sep 10 20:59:54.626477 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.626462 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.648422 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.648394 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 20:59:54.648965 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.648942 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 20:59:54.649156 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.649131 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0b1e8d-c240-4d16-ad26-7e09d88b7da9" path="/var/lib/kubelet/pods/8a0b1e8d-c240-4d16-ad26-7e09d88b7da9/volumes" Sep 10 20:59:54.653204 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.653186 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 20:59:54.653770 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.653741 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 20:59:54.655029 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.655011 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 20:59:54.655124 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.655017 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 20:59:54.656146 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.656132 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 20:59:54.656883 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.656869 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 20:59:54.656982 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.656968 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 20:59:54.657079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.657064 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 20:59:54.657503 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.657489 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 20:59:54.659351 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.659339 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 20:59:54.660668 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.660652 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6fn2g\"" Sep 10 20:59:54.670858 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.670832 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:59:54.673429 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.673413 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 20:59:54.674765 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.674748 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 20:59:54.676193 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.676175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ci5vn4gq6ku2k\"" Sep 10 20:59:54.725699 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d173090-cb95-425d-bb96-16daa5a9f73f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.725841 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.725841 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.725841 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.725841 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725822 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d173090-cb95-425d-bb96-16daa5a9f73f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.725981 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.725981 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.725981 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725913 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.725987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726079 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726255 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v867z\" (UniqueName: \"kubernetes.io/projected/2d173090-cb95-425d-bb96-16daa5a9f73f-kube-api-access-v867z\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726255 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726255 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726255 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726255 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726255 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-config\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.726255 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.726231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.827558 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.827558 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827583 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v867z\" (UniqueName: \"kubernetes.io/projected/2d173090-cb95-425d-bb96-16daa5a9f73f-kube-api-access-v867z\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-config\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.827980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d173090-cb95-425d-bb96-16daa5a9f73f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.828056 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.829345 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d173090-cb95-425d-bb96-16daa5a9f73f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.829345 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.829345 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.829345 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.829345 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.828481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.830498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.829710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.830498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.829756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.830498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.829787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.830498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.829845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.830498 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.830214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.831144 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.831116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.831227 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.831165 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.831399 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.831374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d173090-cb95-425d-bb96-16daa5a9f73f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.831652 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.831626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.832151 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.832125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d173090-cb95-425d-bb96-16daa5a9f73f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.832300 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.832268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d173090-cb95-425d-bb96-16daa5a9f73f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.832905 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.832879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.832995 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.832894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.832995 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.832941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.833519 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.833500 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-config\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.833862 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.833843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.833977 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.833962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.834298 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.834283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.834701 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.834684 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d173090-cb95-425d-bb96-16daa5a9f73f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.852555 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.852525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v867z\" (UniqueName: \"kubernetes.io/projected/2d173090-cb95-425d-bb96-16daa5a9f73f-kube-api-access-v867z\") pod \"prometheus-k8s-0\" (UID: \"2d173090-cb95-425d-bb96-16daa5a9f73f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:54.935525 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:54.935499 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 20:59:55.249279 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:55.249251 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 20:59:55.254066 ip-10-0-132-255 kubenswrapper[2574]: W0910 20:59:55.254041 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d173090_cb95_425d_bb96_16daa5a9f73f.slice/crio-d89d43059974bebdafff043d67e62561747d4f69d2bc831321e4d8e6590f6f85 WatchSource:0}: Error finding container d89d43059974bebdafff043d67e62561747d4f69d2bc831321e4d8e6590f6f85: Status 404 returned error can't find the container with id d89d43059974bebdafff043d67e62561747d4f69d2bc831321e4d8e6590f6f85 Sep 10 20:59:55.529030 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:55.529001 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d173090-cb95-425d-bb96-16daa5a9f73f" containerID="0e8d5b7773f31f5e647c988a1a25391770d0d8f799c4993c0140b5b0b13bbc3f" exitCode=0 Sep 10 20:59:55.529206 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:55.529083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerDied","Data":"0e8d5b7773f31f5e647c988a1a25391770d0d8f799c4993c0140b5b0b13bbc3f"} Sep 10 20:59:55.529206 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:55.529166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerStarted","Data":"d89d43059974bebdafff043d67e62561747d4f69d2bc831321e4d8e6590f6f85"} Sep 10 20:59:56.536889 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.536854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerStarted","Data":"cd533bd55123b91e22171612c8237784ff05b200777f27e57f45157a0a400fc8"} Sep 10 20:59:56.537227 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.536920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerStarted","Data":"77766b2b58d40b529fca88ee52dd266703295d3a149de3c5e7b20edb7579b3b9"} Sep 10 20:59:56.537227 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.536945 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerStarted","Data":"9b515d1e8f39af0a9017ac7c11b4fc3936abc737435b769fe69d148531ba1d11"} Sep 10 20:59:56.537227 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.536969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerStarted","Data":"11415f6da470a06241c496a1568ebbcab9363e4562436cc1db20d840bc033ac3"} Sep 10 20:59:56.537227 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.536992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerStarted","Data":"dee706c3d2ae50e52e3a88e3f6b0d9e99d6de701b725b19a41437c02356d3a82"} Sep 10 20:59:56.537227 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.537016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d173090-cb95-425d-bb96-16daa5a9f73f","Type":"ContainerStarted","Data":"e99e5ca41a120c7ce3176e21644410a9d9dbf07a0fc678087c00f81d229b7158"} Sep 10 20:59:56.591842 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.591777 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.57309294 podStartE2EDuration="2.59176173s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.66879ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:56.589867036 +0000 UTC m=+182.408441158" watchObservedRunningTime="2026-09-10 20:59:56.59176173 +0000 UTC m=+182.410335854" Sep 10 20:59:56.744555 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.744511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:56.873209 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:56.873109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 20:59:59.936574 ip-10-0-132-255 kubenswrapper[2574]: I0910 20:59:59.936538 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 21:00:07.796188 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:07.796156 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:00:07.900709 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:07.900674 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:00:40.203766 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:40.203722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:00:40.339311 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:40.339267 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:00:54.936422 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:54.936342 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 21:00:54.952274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:54.952248 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 21:00:55.771014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:55.770987 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 21:00:58.450488 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:00:58.450455 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:03.590094 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:03.590049 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:03.722586 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:03.722552 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:15.697091 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:15.697043 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:15.812753 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:15.812718 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:24.746911 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:24.746863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:24.851388 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:24.851350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:26.585897 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:26.585868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:31.806448 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:31.806412 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:36.605096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:36.605054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:36.692006 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:36.691980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:45.470159 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:45.470119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:45.567851 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:45.567783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:47.047374 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:47.047340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:47.149354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:47.149320 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:47.230014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:47.229971 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:01:54.558678 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:01:54.558651 2574 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 21:02:07.315500 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:07.315468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:08.896516 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:08.896477 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:11.585154 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:11.585116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:11.683864 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:11.683835 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:13.535346 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:13.535308 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:13.663310 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:13.663279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:13.987057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:13.986980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:14.109605 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:14.109572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:16.239310 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:16.239248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:16.454365 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:16.454333 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:17.937064 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:17.937031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:18.013220 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:18.013185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:24.045577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:24.045541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:24.146436 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:24.146399 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:24.246266 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:24.246231 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:27.737051 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:27.737011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:27.837121 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:27.837086 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:29.938670 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.938643 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj"] Sep 10 21:02:29.941850 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.941825 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:29.944472 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.944449 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 21:02:29.945334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.945315 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-x8bmw\"" Sep 10 21:02:29.945334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.945324 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 21:02:29.953475 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.953454 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj"] Sep 10 21:02:29.975398 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.975368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f4f10-e58b-405a-b237-7e61b93d1d02-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lh2vj\" (UID: \"391f4f10-e58b-405a-b237-7e61b93d1d02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:29.975485 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:29.975414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbhj\" (UniqueName: \"kubernetes.io/projected/391f4f10-e58b-405a-b237-7e61b93d1d02-kube-api-access-8wbhj\") pod \"openshift-lws-operator-fd8ccff4c-lh2vj\" (UID: \"391f4f10-e58b-405a-b237-7e61b93d1d02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:30.076109 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:30.076080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f4f10-e58b-405a-b237-7e61b93d1d02-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lh2vj\" (UID: \"391f4f10-e58b-405a-b237-7e61b93d1d02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:30.076258 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:30.076118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbhj\" (UniqueName: \"kubernetes.io/projected/391f4f10-e58b-405a-b237-7e61b93d1d02-kube-api-access-8wbhj\") pod \"openshift-lws-operator-fd8ccff4c-lh2vj\" (UID: \"391f4f10-e58b-405a-b237-7e61b93d1d02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:30.076460 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:30.076439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f4f10-e58b-405a-b237-7e61b93d1d02-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lh2vj\" (UID: \"391f4f10-e58b-405a-b237-7e61b93d1d02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:30.088456 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:30.088431 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbhj\" (UniqueName: \"kubernetes.io/projected/391f4f10-e58b-405a-b237-7e61b93d1d02-kube-api-access-8wbhj\") pod \"openshift-lws-operator-fd8ccff4c-lh2vj\" (UID: \"391f4f10-e58b-405a-b237-7e61b93d1d02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:30.259274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:30.259200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" Sep 10 21:02:30.553259 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:30.553189 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj"] Sep 10 21:02:30.556601 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:02:30.556552 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod391f4f10_e58b_405a_b237_7e61b93d1d02.slice/crio-7984d705a5eab2581e5947a54af2bc3bcabadd2005f2c46a7a766a3c54ccefa3 WatchSource:0}: Error finding container 7984d705a5eab2581e5947a54af2bc3bcabadd2005f2c46a7a766a3c54ccefa3: Status 404 returned error can't find the container with id 7984d705a5eab2581e5947a54af2bc3bcabadd2005f2c46a7a766a3c54ccefa3 Sep 10 21:02:30.558595 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:30.558579 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 21:02:31.121601 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:31.121565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" event={"ID":"391f4f10-e58b-405a-b237-7e61b93d1d02","Type":"ContainerStarted","Data":"7984d705a5eab2581e5947a54af2bc3bcabadd2005f2c46a7a766a3c54ccefa3"} Sep 10 21:02:33.132045 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:33.131966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" event={"ID":"391f4f10-e58b-405a-b237-7e61b93d1d02","Type":"ContainerStarted","Data":"690fe2403833e01f7e9bd7d5f8f1f896eaea8868bb01fc868208ff13bd85c327"} Sep 10 21:02:33.181943 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:33.181894 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lh2vj" podStartSLOduration=1.910252353 podStartE2EDuration="4.181878351s" totalImagesPullingTime="2.271625998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:33.180705024 +0000 UTC m=+338.999279146" watchObservedRunningTime="2026-09-10 21:02:33.181878351 +0000 UTC m=+339.000452472" Sep 10 21:02:33.888546 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:33.888509 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:33.963296 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:33.963263 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:35.831878 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:35.831836 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:47.722510 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:47.722429 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:47.869270 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:47.869228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:48.266390 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.266355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:48.624006 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.623970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:48.733727 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.733668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:48.810798 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.810768 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt"] Sep 10 21:02:48.814271 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.814256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:48.817738 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.817711 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 21:02:48.817924 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.817899 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wgnwx\"" Sep 10 21:02:48.818155 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.818138 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 21:02:48.818344 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.818330 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 21:02:48.820281 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.820264 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 21:02:48.853822 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.853792 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt"] Sep 10 21:02:48.918794 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.918715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ca938d6-275e-41dc-9815-2577f9101c39-manifests\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:48.918794 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.918751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ca938d6-275e-41dc-9815-2577f9101c39-tmp\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:48.918794 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.918790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ca938d6-275e-41dc-9815-2577f9101c39-webhook-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:48.919016 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.918873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ca938d6-275e-41dc-9815-2577f9101c39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:48.919016 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:48.918951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449gq\" (UniqueName: \"kubernetes.io/projected/0ca938d6-275e-41dc-9815-2577f9101c39-kube-api-access-449gq\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.020111 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.020075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ca938d6-275e-41dc-9815-2577f9101c39-tmp\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.020281 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.020131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ca938d6-275e-41dc-9815-2577f9101c39-webhook-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.020281 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.020176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ca938d6-275e-41dc-9815-2577f9101c39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.020281 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.020218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449gq\" (UniqueName: \"kubernetes.io/projected/0ca938d6-275e-41dc-9815-2577f9101c39-kube-api-access-449gq\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.020281 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.020246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ca938d6-275e-41dc-9815-2577f9101c39-manifests\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.020606 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.020586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ca938d6-275e-41dc-9815-2577f9101c39-manifests\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.022487 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.022463 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ca938d6-275e-41dc-9815-2577f9101c39-tmp\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.022935 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.022914 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ca938d6-275e-41dc-9815-2577f9101c39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.022985 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.022914 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ca938d6-275e-41dc-9815-2577f9101c39-webhook-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.031669 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.031643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-449gq\" (UniqueName: \"kubernetes.io/projected/0ca938d6-275e-41dc-9815-2577f9101c39-kube-api-access-449gq\") pod \"opendatahub-operator-controller-manager-769bdd6b4-rbzpt\" (UID: \"0ca938d6-275e-41dc-9815-2577f9101c39\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.124235 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.124204 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:49.151410 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.151379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:49.245673 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.245639 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:49.326854 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.326781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:49.396703 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.396676 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:49.412517 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.412491 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt"] Sep 10 21:02:49.415719 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:02:49.415697 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca938d6_275e_41dc_9815_2577f9101c39.slice/crio-7dcefc81c60ba1aa8d955956d51fed3de680339639872cd3bf1119823653ece5 WatchSource:0}: Error finding container 7dcefc81c60ba1aa8d955956d51fed3de680339639872cd3bf1119823653ece5: Status 404 returned error can't find the container with id 7dcefc81c60ba1aa8d955956d51fed3de680339639872cd3bf1119823653ece5 Sep 10 21:02:49.506559 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.506477 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:49.616788 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:49.616740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:50.205621 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:50.205582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" event={"ID":"0ca938d6-275e-41dc-9815-2577f9101c39","Type":"ContainerStarted","Data":"7dcefc81c60ba1aa8d955956d51fed3de680339639872cd3bf1119823653ece5"} Sep 10 21:02:50.207490 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:50.207422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:50.315957 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:50.315921 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:53.220493 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:53.220458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" event={"ID":"0ca938d6-275e-41dc-9815-2577f9101c39","Type":"ContainerStarted","Data":"c3944c22535108db946548bb31edb36881baf2010b6d2a5eb5f864ea10fe2be9"} Sep 10 21:02:54.226101 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:54.226069 2574 generic.go:356] "Generic (PLEG): container finished" podID="0ca938d6-275e-41dc-9815-2577f9101c39" containerID="c3944c22535108db946548bb31edb36881baf2010b6d2a5eb5f864ea10fe2be9" exitCode=0 Sep 10 21:02:54.226481 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:54.226140 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" event={"ID":"0ca938d6-275e-41dc-9815-2577f9101c39","Type":"ContainerDied","Data":"c3944c22535108db946548bb31edb36881baf2010b6d2a5eb5f864ea10fe2be9"} Sep 10 21:02:55.237887 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:55.237845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" event={"ID":"0ca938d6-275e-41dc-9815-2577f9101c39","Type":"ContainerStarted","Data":"a3c70a8b69fd2535655a0567aab2a0da2ea47fe1fe07f5c2a4e183bee03ddf2b"} Sep 10 21:02:55.238290 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:55.238037 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:02:55.267220 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:55.267152 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" podStartSLOduration=3.083959808 podStartE2EDuration="7.267138203s" totalImagesPullingTime="3.18099142s" totalInitContainerRuntime="1.002186975s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:55.266962353 +0000 UTC m=+361.085536476" watchObservedRunningTime="2026-09-10 21:02:55.267138203 +0000 UTC m=+361.085712323" Sep 10 21:02:58.403244 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:58.403208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:02:58.468712 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:02:58.468682 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:06.244423 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:06.244391 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-rbzpt" Sep 10 21:03:07.961334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:07.961287 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:08.125861 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.125821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:08.364343 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.364303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:08.780422 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.780394 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9"] Sep 10 21:03:08.804090 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.804054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:08.918309 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.918279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9"] Sep 10 21:03:08.918475 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.918372 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:08.921643 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.921617 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 21:03:08.922641 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.922624 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 21:03:08.922721 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.922655 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 21:03:08.922721 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.922629 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lwtnn\"" Sep 10 21:03:08.989048 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.989020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c66153f-4ac5-4304-ac05-b3f32173e600-metrics-cert\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:08.989506 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.989068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkvg\" (UniqueName: \"kubernetes.io/projected/6c66153f-4ac5-4304-ac05-b3f32173e600-kube-api-access-ldkvg\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:08.989506 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.989115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c66153f-4ac5-4304-ac05-b3f32173e600-cert\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:08.989506 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:08.989144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c66153f-4ac5-4304-ac05-b3f32173e600-manager-config\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.090054 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.089973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c66153f-4ac5-4304-ac05-b3f32173e600-metrics-cert\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.090054 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.090025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkvg\" (UniqueName: \"kubernetes.io/projected/6c66153f-4ac5-4304-ac05-b3f32173e600-kube-api-access-ldkvg\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.090054 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.090052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c66153f-4ac5-4304-ac05-b3f32173e600-cert\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.090281 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.090093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c66153f-4ac5-4304-ac05-b3f32173e600-manager-config\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.090750 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.090725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c66153f-4ac5-4304-ac05-b3f32173e600-manager-config\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.092605 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.092583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c66153f-4ac5-4304-ac05-b3f32173e600-cert\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.092709 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.092583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c66153f-4ac5-4304-ac05-b3f32173e600-metrics-cert\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.102079 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.102050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkvg\" (UniqueName: \"kubernetes.io/projected/6c66153f-4ac5-4304-ac05-b3f32173e600-kube-api-access-ldkvg\") pod \"lws-controller-manager-799dfcfb97-kpph9\" (UID: \"6c66153f-4ac5-4304-ac05-b3f32173e600\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.227259 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.227227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:09.508355 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:09.508332 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9"] Sep 10 21:03:09.510643 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:03:09.510618 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c66153f_4ac5_4304_ac05_b3f32173e600.slice/crio-3d164b644ff358cb7116ff5e9a8b86f738680f76e5e8ece114c8c7fb4caff394 WatchSource:0}: Error finding container 3d164b644ff358cb7116ff5e9a8b86f738680f76e5e8ece114c8c7fb4caff394: Status 404 returned error can't find the container with id 3d164b644ff358cb7116ff5e9a8b86f738680f76e5e8ece114c8c7fb4caff394 Sep 10 21:03:10.308862 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:10.308825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" event={"ID":"6c66153f-4ac5-4304-ac05-b3f32173e600","Type":"ContainerStarted","Data":"3d164b644ff358cb7116ff5e9a8b86f738680f76e5e8ece114c8c7fb4caff394"} Sep 10 21:03:10.386667 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:10.386632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:10.461091 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:10.461054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:11.315025 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:11.314995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" event={"ID":"6c66153f-4ac5-4304-ac05-b3f32173e600","Type":"ContainerStarted","Data":"9ad9ccdb0f2253fa6766872ae08bfaf8d051e71f1140309f62aad06c40759987"} Sep 10 21:03:11.315468 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:11.315054 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:11.332502 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:11.332450 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" podStartSLOduration=2.008636468 podStartE2EDuration="3.332429834s" totalImagesPullingTime="1.323793366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:11.33189596 +0000 UTC m=+377.150470081" watchObservedRunningTime="2026-09-10 21:03:11.332429834 +0000 UTC m=+377.151003958" Sep 10 21:03:11.798309 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:11.798274 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:15.737542 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:15.737502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:15.836645 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:15.836614 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:16.041849 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.041803 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9"] Sep 10 21:03:16.044065 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.044050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.049913 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.049891 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 21:03:16.050651 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.050631 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 21:03:16.050740 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.050683 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 21:03:16.050740 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.050721 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sbd9l\"" Sep 10 21:03:16.059788 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.059763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9"] Sep 10 21:03:16.104421 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.104390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:16.153450 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153590 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153590 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153590 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153590 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153590 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153765 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153765 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblt2\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-kube-api-access-tblt2\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.153765 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.153684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.156033 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.156010 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc"] Sep 10 21:03:16.158419 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.158404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.237515 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.237480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc"] Sep 10 21:03:16.254060 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254223 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.254223 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgz2t\" (UniqueName: \"kubernetes.io/projected/61512794-0fb8-4fa7-be7e-50c6a850b78e-kube-api-access-tgz2t\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254223 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254223 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254223 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblt2\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-kube-api-access-tblt2\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61512794-0fb8-4fa7-be7e-50c6a850b78e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.254497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.255096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.254998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255493 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255261 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255493 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255450 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255600 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.255600 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.255527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.256777 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.256758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.256921 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.256905 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.273443 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.273420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblt2\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-kube-api-access-tblt2\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.273860 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.273843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.355070 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.354996 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:16.356255 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgz2t\" (UniqueName: \"kubernetes.io/projected/61512794-0fb8-4fa7-be7e-50c6a850b78e-kube-api-access-tgz2t\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356379 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356379 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356379 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356379 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356750 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356750 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61512794-0fb8-4fa7-be7e-50c6a850b78e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356750 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356750 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356985 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356833 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356985 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356842 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.356985 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.356900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.357286 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.357241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61512794-0fb8-4fa7-be7e-50c6a850b78e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.357286 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.357257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.359080 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.359054 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.359513 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.359491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.367919 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.367898 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgz2t\" (UniqueName: \"kubernetes.io/projected/61512794-0fb8-4fa7-be7e-50c6a850b78e-kube-api-access-tgz2t\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.372157 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.372133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:16.372516 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.372496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61512794-0fb8-4fa7-be7e-50c6a850b78e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc\" (UID: \"61512794-0fb8-4fa7-be7e-50c6a850b78e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.470376 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.470344 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:16.541150 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.541109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:16.651138 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.650965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9"] Sep 10 21:03:16.653265 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:03:16.653238 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca9b17a_1bdf_426b_9140_ae7ca50cd01d.slice/crio-fec0e57f8c88d8c8c06391cfb8e515b15675e6941f21d017aa348dda01594a14 WatchSource:0}: Error finding container fec0e57f8c88d8c8c06391cfb8e515b15675e6941f21d017aa348dda01594a14: Status 404 returned error can't find the container with id fec0e57f8c88d8c8c06391cfb8e515b15675e6941f21d017aa348dda01594a14 Sep 10 21:03:16.751530 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:16.751504 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc"] Sep 10 21:03:16.754048 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:03:16.754021 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61512794_0fb8_4fa7_be7e_50c6a850b78e.slice/crio-5f71e856eb84c8f950b76193a2b8f2cb5b1bce139a6a497db93cf5bd457170bc WatchSource:0}: Error finding container 5f71e856eb84c8f950b76193a2b8f2cb5b1bce139a6a497db93cf5bd457170bc: Status 404 returned error can't find the container with id 5f71e856eb84c8f950b76193a2b8f2cb5b1bce139a6a497db93cf5bd457170bc Sep 10 21:03:17.358052 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:17.357960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" event={"ID":"eca9b17a-1bdf-426b-9140-ae7ca50cd01d","Type":"ContainerStarted","Data":"fec0e57f8c88d8c8c06391cfb8e515b15675e6941f21d017aa348dda01594a14"} Sep 10 21:03:17.359608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:17.359571 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" event={"ID":"61512794-0fb8-4fa7-be7e-50c6a850b78e","Type":"ContainerStarted","Data":"5f71e856eb84c8f950b76193a2b8f2cb5b1bce139a6a497db93cf5bd457170bc"} Sep 10 21:03:19.347607 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.347569 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jgtss"] Sep 10 21:03:19.355264 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.355236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:19.358209 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.358185 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 21:03:19.358736 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.358696 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qs5m7\"" Sep 10 21:03:19.377217 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.376844 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jgtss"] Sep 10 21:03:19.418408 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.418340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:19.488216 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.488166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-cert\") pod \"odh-model-controller-7998c67dc4-jgtss\" (UID: \"91fa8523-3c84-41ba-a873-5b1fe5bf3f68\") " pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:19.488389 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.488362 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6q4m\" (UniqueName: \"kubernetes.io/projected/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-kube-api-access-w6q4m\") pod \"odh-model-controller-7998c67dc4-jgtss\" (UID: \"91fa8523-3c84-41ba-a873-5b1fe5bf3f68\") " pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:19.517364 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.517337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:19.518459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.518410 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:03:19.518556 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.518506 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:03:19.518556 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.518548 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:03:19.532154 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.532123 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:03:19.532238 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.532194 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:03:19.532238 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.532225 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:03:19.588855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.588828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6q4m\" (UniqueName: \"kubernetes.io/projected/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-kube-api-access-w6q4m\") pod \"odh-model-controller-7998c67dc4-jgtss\" (UID: \"91fa8523-3c84-41ba-a873-5b1fe5bf3f68\") " pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:19.588944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.588889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-cert\") pod \"odh-model-controller-7998c67dc4-jgtss\" (UID: \"91fa8523-3c84-41ba-a873-5b1fe5bf3f68\") " pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:19.589053 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:03:19.589036 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 21:03:19.589128 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:03:19.589112 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-cert podName:91fa8523-3c84-41ba-a873-5b1fe5bf3f68 nodeName:}" failed. No retries permitted until 2026-09-10 21:03:20.089091495 +0000 UTC m=+385.907665611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-cert") pod "odh-model-controller-7998c67dc4-jgtss" (UID: "91fa8523-3c84-41ba-a873-5b1fe5bf3f68") : secret "odh-model-controller-webhook-cert" not found Sep 10 21:03:19.606471 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.606410 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6q4m\" (UniqueName: \"kubernetes.io/projected/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-kube-api-access-w6q4m\") pod \"odh-model-controller-7998c67dc4-jgtss\" (UID: \"91fa8523-3c84-41ba-a873-5b1fe5bf3f68\") " pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:19.641477 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.641437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:19.854114 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:19.854078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:20.034167 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.034137 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-txpqf"] Sep 10 21:03:20.037136 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.037113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.039439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.039414 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 21:03:20.048585 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.048553 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-txpqf"] Sep 10 21:03:20.056692 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.056666 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:20.092739 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.092714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-cert\") pod \"odh-model-controller-7998c67dc4-jgtss\" (UID: \"91fa8523-3c84-41ba-a873-5b1fe5bf3f68\") " pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:20.095302 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.095272 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91fa8523-3c84-41ba-a873-5b1fe5bf3f68-cert\") pod \"odh-model-controller-7998c67dc4-jgtss\" (UID: \"91fa8523-3c84-41ba-a873-5b1fe5bf3f68\") " pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:20.193635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.193593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9npk\" (UniqueName: \"kubernetes.io/projected/7c189779-ccac-40cb-87fb-01dd5dd261c4-kube-api-access-b9npk\") pod \"model-serving-api-54d89bd5fc-txpqf\" (UID: \"7c189779-ccac-40cb-87fb-01dd5dd261c4\") " pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.193834 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.193666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c189779-ccac-40cb-87fb-01dd5dd261c4-tls-certs\") pod \"model-serving-api-54d89bd5fc-txpqf\" (UID: \"7c189779-ccac-40cb-87fb-01dd5dd261c4\") " pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.285944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.285859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:20.295063 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.295031 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c189779-ccac-40cb-87fb-01dd5dd261c4-tls-certs\") pod \"model-serving-api-54d89bd5fc-txpqf\" (UID: \"7c189779-ccac-40cb-87fb-01dd5dd261c4\") " pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.295171 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.295135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9npk\" (UniqueName: \"kubernetes.io/projected/7c189779-ccac-40cb-87fb-01dd5dd261c4-kube-api-access-b9npk\") pod \"model-serving-api-54d89bd5fc-txpqf\" (UID: \"7c189779-ccac-40cb-87fb-01dd5dd261c4\") " pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.295225 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:03:20.295202 2574 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 21:03:20.295282 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:03:20.295271 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c189779-ccac-40cb-87fb-01dd5dd261c4-tls-certs podName:7c189779-ccac-40cb-87fb-01dd5dd261c4 nodeName:}" failed. No retries permitted until 2026-09-10 21:03:20.795250879 +0000 UTC m=+386.613824984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c189779-ccac-40cb-87fb-01dd5dd261c4-tls-certs") pod "model-serving-api-54d89bd5fc-txpqf" (UID: "7c189779-ccac-40cb-87fb-01dd5dd261c4") : secret "model-serving-api-tls" not found Sep 10 21:03:20.305445 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.305417 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9npk\" (UniqueName: \"kubernetes.io/projected/7c189779-ccac-40cb-87fb-01dd5dd261c4-kube-api-access-b9npk\") pod \"model-serving-api-54d89bd5fc-txpqf\" (UID: \"7c189779-ccac-40cb-87fb-01dd5dd261c4\") " pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.391546 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.391510 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" event={"ID":"eca9b17a-1bdf-426b-9140-ae7ca50cd01d","Type":"ContainerStarted","Data":"10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701"} Sep 10 21:03:20.393331 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.393297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" event={"ID":"61512794-0fb8-4fa7-be7e-50c6a850b78e","Type":"ContainerStarted","Data":"66a0cf4136eaf67d0b7ad61c17c8d6a72044fab5a1151e00e2cc090d541b24fe"} Sep 10 21:03:20.411696 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.411642 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" podStartSLOduration=2.548662776 podStartE2EDuration="5.411622158s" totalImagesPullingTime="2.862959382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:20.410554397 +0000 UTC m=+386.229128520" watchObservedRunningTime="2026-09-10 21:03:20.411622158 +0000 UTC m=+386.230196290" Sep 10 21:03:20.437981 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.437922 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" podStartSLOduration=1.662084795 podStartE2EDuration="4.437901554s" totalImagesPullingTime="2.775816759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:20.436564444 +0000 UTC m=+386.255138566" watchObservedRunningTime="2026-09-10 21:03:20.437901554 +0000 UTC m=+386.256475677" Sep 10 21:03:20.470782 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.470744 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:20.475968 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.475944 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:20.563222 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.563193 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jgtss"] Sep 10 21:03:20.565684 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:03:20.565639 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fa8523_3c84_41ba_a873_5b1fe5bf3f68.slice/crio-71b309756a4ffa448dcc64e867b283e0ff6d76898430dd90c7f864bee89e58bf WatchSource:0}: Error finding container 71b309756a4ffa448dcc64e867b283e0ff6d76898430dd90c7f864bee89e58bf: Status 404 returned error can't find the container with id 71b309756a4ffa448dcc64e867b283e0ff6d76898430dd90c7f864bee89e58bf Sep 10 21:03:20.800230 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.800186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c189779-ccac-40cb-87fb-01dd5dd261c4-tls-certs\") pod \"model-serving-api-54d89bd5fc-txpqf\" (UID: \"7c189779-ccac-40cb-87fb-01dd5dd261c4\") " pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.802646 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.802623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c189779-ccac-40cb-87fb-01dd5dd261c4-tls-certs\") pod \"model-serving-api-54d89bd5fc-txpqf\" (UID: \"7c189779-ccac-40cb-87fb-01dd5dd261c4\") " pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:20.852988 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.852886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:20.950192 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:20.950152 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:21.234761 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.234676 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-txpqf"] Sep 10 21:03:21.356862 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.356251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:21.358577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.358248 2574 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 10 21:03:21.358577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.358320 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 10 21:03:21.402509 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.402475 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" event={"ID":"91fa8523-3c84-41ba-a873-5b1fe5bf3f68","Type":"ContainerStarted","Data":"71b309756a4ffa448dcc64e867b283e0ff6d76898430dd90c7f864bee89e58bf"} Sep 10 21:03:21.404972 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.404849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" event={"ID":"7c189779-ccac-40cb-87fb-01dd5dd261c4","Type":"ContainerStarted","Data":"1f33c1c480aca3adb749946078c7752a9c4877a95739f400896624faf20e8855"} Sep 10 21:03:21.405284 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.404913 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:21.406078 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.406036 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc" Sep 10 21:03:21.470002 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:21.469903 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9"] Sep 10 21:03:22.322617 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:22.322590 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kpph9" Sep 10 21:03:22.356342 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:22.356184 2574 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 10 21:03:22.356342 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:22.356253 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 10 21:03:23.356117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:23.355680 2574 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 10 21:03:23.356117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:23.355742 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 10 21:03:23.417749 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:23.417662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" event={"ID":"91fa8523-3c84-41ba-a873-5b1fe5bf3f68","Type":"ContainerStarted","Data":"2c111929a350b8c96efaf35a0822e1dd58f6e96eba87d2be75c735f5ca02fc0d"} Sep 10 21:03:23.417982 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:23.417961 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:23.418146 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:23.418116 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerName="istio-proxy" containerID="cri-o://10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701" gracePeriod=30 Sep 10 21:03:23.444835 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:23.444764 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" podStartSLOduration=1.851051223 podStartE2EDuration="4.444742881s" totalImagesPullingTime="2.593691658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:23.441372098 +0000 UTC m=+389.259946220" watchObservedRunningTime="2026-09-10 21:03:23.444742881 +0000 UTC m=+389.263317002" Sep 10 21:03:25.429209 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:25.429123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" event={"ID":"7c189779-ccac-40cb-87fb-01dd5dd261c4","Type":"ContainerStarted","Data":"7ef4b778640fb218cf4b88b62513c30bce9e6db5cf5b2001cc580aeeef0877b5"} Sep 10 21:03:25.429600 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:25.429219 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:25.450825 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:25.450747 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" podStartSLOduration=1.522836474 podStartE2EDuration="5.450725885s" totalImagesPullingTime="3.927889411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:25.448595155 +0000 UTC m=+391.267169288" watchObservedRunningTime="2026-09-10 21:03:25.450725885 +0000 UTC m=+391.269300007" Sep 10 21:03:28.665736 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.665713 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:28.673509 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673478 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-socket\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673519 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-podinfo\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673541 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-token\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673560 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tblt2\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-kube-api-access-tblt2\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673599 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-credential-socket\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673621 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-envoy\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673925 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673662 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-data\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673925 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673696 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istiod-ca-cert\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.673925 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:03:28.673925 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.673722 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-certs\") pod \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\" (UID: \"eca9b17a-1bdf-426b-9140-ae7ca50cd01d\") " Sep 10 21:03:28.674131 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.674021 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-data" (OuterVolumeSpecName: "istio-data") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:03:28.674131 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.674092 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-data\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.674131 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.674114 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-socket\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.674285 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.674179 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:03:28.674340 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.674325 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:03:28.674397 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.674328 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:03:28.676206 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.676148 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:03:28.676300 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.676199 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-kube-api-access-tblt2" (OuterVolumeSpecName: "kube-api-access-tblt2") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "kube-api-access-tblt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:03:28.676300 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.676262 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-token" (OuterVolumeSpecName: "istio-token") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:03:28.676300 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.676261 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eca9b17a-1bdf-426b-9140-ae7ca50cd01d" (UID: "eca9b17a-1bdf-426b-9140-ae7ca50cd01d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:03:28.774824 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.774783 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-envoy\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.774951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.774834 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istiod-ca-cert\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.774951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.774848 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-workload-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.774951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.774861 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-podinfo\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.774951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.774872 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-istio-token\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.774951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.774880 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblt2\" (UniqueName: \"kubernetes.io/projected/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-kube-api-access-tblt2\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:28.774951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:28.774889 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca9b17a-1bdf-426b-9140-ae7ca50cd01d-credential-socket\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:03:29.452278 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.452250 2574 generic.go:356] "Generic (PLEG): container finished" podID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerID="10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701" exitCode=0 Sep 10 21:03:29.452450 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.452299 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" Sep 10 21:03:29.452450 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.452339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" event={"ID":"eca9b17a-1bdf-426b-9140-ae7ca50cd01d","Type":"ContainerDied","Data":"10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701"} Sep 10 21:03:29.452450 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.452391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9" event={"ID":"eca9b17a-1bdf-426b-9140-ae7ca50cd01d","Type":"ContainerDied","Data":"fec0e57f8c88d8c8c06391cfb8e515b15675e6941f21d017aa348dda01594a14"} Sep 10 21:03:29.452450 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.452415 2574 scope.go:117] "RemoveContainer" containerID="10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701" Sep 10 21:03:29.463038 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.463008 2574 scope.go:117] "RemoveContainer" containerID="10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701" Sep 10 21:03:29.463516 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:03:29.463489 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701\": container with ID starting with 10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701 not found: ID does not exist" containerID="10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701" Sep 10 21:03:29.463622 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.463521 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701"} err="failed to get container status \"10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701\": rpc error: code = NotFound desc = could not find container \"10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701\": container with ID starting with 10d1e2e3730fa454c413434a0b9af379d6764ce8258d86fc3340be539b573701 not found: ID does not exist" Sep 10 21:03:29.479694 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.479666 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9"] Sep 10 21:03:29.493712 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:29.493685 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcfk9t9"] Sep 10 21:03:30.650042 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:30.650010 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" path="/var/lib/kubelet/pods/eca9b17a-1bdf-426b-9140-ae7ca50cd01d/volumes" Sep 10 21:03:31.207849 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.207796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:31.292045 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.292008 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5qckg"] Sep 10 21:03:31.292939 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.292915 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerName="istio-proxy" Sep 10 21:03:31.293100 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.293089 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerName="istio-proxy" Sep 10 21:03:31.293279 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.293268 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca9b17a-1bdf-426b-9140-ae7ca50cd01d" containerName="istio-proxy" Sep 10 21:03:31.302053 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.302029 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl"] Sep 10 21:03:31.302210 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.302123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.304595 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.304572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:31.305786 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.305746 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 21:03:31.305970 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.305928 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9kp5c\"" Sep 10 21:03:31.306795 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.306778 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rntht\"" Sep 10 21:03:31.308857 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.308770 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 21:03:31.311725 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.311706 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5qckg"] Sep 10 21:03:31.320200 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.320179 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl"] Sep 10 21:03:31.342572 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.342534 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:31.395082 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.395060 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cc26d22-85b6-4d2e-a046-f665fc4fb3bb-cert\") pod \"kserve-controller-manager-d6f4b65b8-5qckg\" (UID: \"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.395222 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.395119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crn9\" (UniqueName: \"kubernetes.io/projected/8cc26d22-85b6-4d2e-a046-f665fc4fb3bb-kube-api-access-6crn9\") pod \"kserve-controller-manager-d6f4b65b8-5qckg\" (UID: \"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.395222 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.395154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjsk\" (UniqueName: \"kubernetes.io/projected/e11d85cb-a02f-4982-a9b8-a69462a98e71-kube-api-access-8qjsk\") pod \"llmisvc-controller-manager-777c6bd7d5-zd8bl\" (UID: \"e11d85cb-a02f-4982-a9b8-a69462a98e71\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:31.395222 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.395184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e11d85cb-a02f-4982-a9b8-a69462a98e71-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zd8bl\" (UID: \"e11d85cb-a02f-4982-a9b8-a69462a98e71\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:31.428459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.428425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:31.496183 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.496104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjsk\" (UniqueName: \"kubernetes.io/projected/e11d85cb-a02f-4982-a9b8-a69462a98e71-kube-api-access-8qjsk\") pod \"llmisvc-controller-manager-777c6bd7d5-zd8bl\" (UID: \"e11d85cb-a02f-4982-a9b8-a69462a98e71\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:31.496183 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.496162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e11d85cb-a02f-4982-a9b8-a69462a98e71-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zd8bl\" (UID: \"e11d85cb-a02f-4982-a9b8-a69462a98e71\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:31.496388 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.496206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cc26d22-85b6-4d2e-a046-f665fc4fb3bb-cert\") pod \"kserve-controller-manager-d6f4b65b8-5qckg\" (UID: \"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.496388 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:03:31.496247 2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 10 21:03:31.496388 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.496286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crn9\" (UniqueName: \"kubernetes.io/projected/8cc26d22-85b6-4d2e-a046-f665fc4fb3bb-kube-api-access-6crn9\") pod \"kserve-controller-manager-d6f4b65b8-5qckg\" (UID: \"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.496388 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:03:31.496303 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11d85cb-a02f-4982-a9b8-a69462a98e71-cert podName:e11d85cb-a02f-4982-a9b8-a69462a98e71 nodeName:}" failed. No retries permitted until 2026-09-10 21:03:31.996285193 +0000 UTC m=+397.814859298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e11d85cb-a02f-4982-a9b8-a69462a98e71-cert") pod "llmisvc-controller-manager-777c6bd7d5-zd8bl" (UID: "e11d85cb-a02f-4982-a9b8-a69462a98e71") : secret "llmisvc-webhook-server-cert" not found Sep 10 21:03:31.498824 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.498788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cc26d22-85b6-4d2e-a046-f665fc4fb3bb-cert\") pod \"kserve-controller-manager-d6f4b65b8-5qckg\" (UID: \"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.506771 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.506751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjsk\" (UniqueName: \"kubernetes.io/projected/e11d85cb-a02f-4982-a9b8-a69462a98e71-kube-api-access-8qjsk\") pod \"llmisvc-controller-manager-777c6bd7d5-zd8bl\" (UID: \"e11d85cb-a02f-4982-a9b8-a69462a98e71\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:31.515020 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.515002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crn9\" (UniqueName: \"kubernetes.io/projected/8cc26d22-85b6-4d2e-a046-f665fc4fb3bb-kube-api-access-6crn9\") pod \"kserve-controller-manager-d6f4b65b8-5qckg\" (UID: \"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.620502 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.620465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:31.811930 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.811896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:31.930998 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.930918 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5qckg"] Sep 10 21:03:31.933666 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:03:31.933643 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc26d22_85b6_4d2e_a046_f665fc4fb3bb.slice/crio-b10837649a3f664f757ea2724ca62b17b195f54a6970793ae78a98e43e00fb5c WatchSource:0}: Error finding container b10837649a3f664f757ea2724ca62b17b195f54a6970793ae78a98e43e00fb5c: Status 404 returned error can't find the container with id b10837649a3f664f757ea2724ca62b17b195f54a6970793ae78a98e43e00fb5c Sep 10 21:03:31.999997 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:31.999963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e11d85cb-a02f-4982-a9b8-a69462a98e71-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zd8bl\" (UID: \"e11d85cb-a02f-4982-a9b8-a69462a98e71\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:32.002464 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:32.002445 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e11d85cb-a02f-4982-a9b8-a69462a98e71-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zd8bl\" (UID: \"e11d85cb-a02f-4982-a9b8-a69462a98e71\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:32.228565 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:32.228481 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:32.468959 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:32.468905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" event={"ID":"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb","Type":"ContainerStarted","Data":"b10837649a3f664f757ea2724ca62b17b195f54a6970793ae78a98e43e00fb5c"} Sep 10 21:03:32.511201 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:32.511127 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl"] Sep 10 21:03:32.513970 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:03:32.513940 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode11d85cb_a02f_4982_a9b8_a69462a98e71.slice/crio-a84c83ccac93f2be6982f7d27344c5957c256c2a9d73ff491d7cb5418547fcd2 WatchSource:0}: Error finding container a84c83ccac93f2be6982f7d27344c5957c256c2a9d73ff491d7cb5418547fcd2: Status 404 returned error can't find the container with id a84c83ccac93f2be6982f7d27344c5957c256c2a9d73ff491d7cb5418547fcd2 Sep 10 21:03:32.615507 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:32.615480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:32.739355 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:32.739318 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:32.845656 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:32.845622 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:33.480433 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:33.480390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" event={"ID":"e11d85cb-a02f-4982-a9b8-a69462a98e71","Type":"ContainerStarted","Data":"a84c83ccac93f2be6982f7d27344c5957c256c2a9d73ff491d7cb5418547fcd2"} Sep 10 21:03:34.426191 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:34.426155 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jgtss" Sep 10 21:03:35.495609 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:35.495578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" event={"ID":"8cc26d22-85b6-4d2e-a046-f665fc4fb3bb","Type":"ContainerStarted","Data":"8f7f39d7e1c63a030c3745d06f43a39f16275fc9ed9b975b440257916bfb5105"} Sep 10 21:03:35.496032 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:35.495694 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:03:35.497184 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:35.497165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" event={"ID":"e11d85cb-a02f-4982-a9b8-a69462a98e71","Type":"ContainerStarted","Data":"a1867f8eeeeae66e7bc7eb678b69de0e938eb37bc1598eba8e9c35afcc7cccb5"} Sep 10 21:03:35.497302 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:35.497288 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:03:35.521325 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:35.521285 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" podStartSLOduration=1.5353547220000001 podStartE2EDuration="4.521273869s" totalImagesPullingTime="2.985919147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:35.519162605 +0000 UTC m=+401.337736731" watchObservedRunningTime="2026-09-10 21:03:35.521273869 +0000 UTC m=+401.339847989" Sep 10 21:03:35.540780 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:35.540731 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" podStartSLOduration=2.132565788 podStartE2EDuration="4.54071423s" totalImagesPullingTime="2.408148442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:35.538076244 +0000 UTC m=+401.356650364" watchObservedRunningTime="2026-09-10 21:03:35.54071423 +0000 UTC m=+401.359288354" Sep 10 21:03:36.440389 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:36.440360 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-txpqf" Sep 10 21:03:54.765363 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:54.765326 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:54.829374 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:54.829335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:54.915614 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:54.915535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:55.040849 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:55.040792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:55.507224 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:55.507192 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:03:55.614791 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:03:55.614761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:05.891719 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:05.891683 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:05.950489 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:05.950452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:06.507711 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:06.507684 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zd8bl" Sep 10 21:04:06.510573 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:06.510549 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5qckg" Sep 10 21:04:07.269578 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:07.269541 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:20.136114 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:20.136078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:20.235588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:20.235551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:32.570263 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:32.570226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:33.360647 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:33.360604 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:36.594018 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:36.593988 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:36.813530 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:36.813486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:36.926630 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:36.926524 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:38.120033 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.120000 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp"] Sep 10 21:04:38.124871 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.124851 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" Sep 10 21:04:38.128113 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.128093 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 21:04:38.128113 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.128107 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 21:04:38.128442 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.128428 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 21:04:38.130576 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.130560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nmmwc\"" Sep 10 21:04:38.138480 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.138460 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp"] Sep 10 21:04:38.173483 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.173455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbfl\" (UniqueName: \"kubernetes.io/projected/f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018-kube-api-access-vkbfl\") pod \"dns-operator-controller-manager-7b84cd7c9c-lh6qp\" (UID: \"f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" Sep 10 21:04:38.254126 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.254090 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:38.274148 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.274122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbfl\" (UniqueName: \"kubernetes.io/projected/f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018-kube-api-access-vkbfl\") pod \"dns-operator-controller-manager-7b84cd7c9c-lh6qp\" (UID: \"f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" Sep 10 21:04:38.285394 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.285369 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbfl\" (UniqueName: \"kubernetes.io/projected/f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018-kube-api-access-vkbfl\") pod \"dns-operator-controller-manager-7b84cd7c9c-lh6qp\" (UID: \"f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" Sep 10 21:04:38.435375 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.435291 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" Sep 10 21:04:38.713285 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.713260 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp"] Sep 10 21:04:38.715371 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:04:38.715347 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d8fe45_2d2e_44a9_bb6a_bf3c2127f018.slice/crio-d87ade3ec30403567a2e5519cd2f0c290310aca5d5a865e69ac93f804c9cd1c9 WatchSource:0}: Error finding container d87ade3ec30403567a2e5519cd2f0c290310aca5d5a865e69ac93f804c9cd1c9: Status 404 returned error can't find the container with id d87ade3ec30403567a2e5519cd2f0c290310aca5d5a865e69ac93f804c9cd1c9 Sep 10 21:04:38.783462 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.783440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" event={"ID":"f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018","Type":"ContainerStarted","Data":"d87ade3ec30403567a2e5519cd2f0c290310aca5d5a865e69ac93f804c9cd1c9"} Sep 10 21:04:38.981663 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:38.981573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:39.791534 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:39.791494 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:39.918994 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:39.918953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:40.021308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.021278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:40.585971 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.585929 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n"] Sep 10 21:04:40.597186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.597159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:04:40.601740 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.601717 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xnlw7\"" Sep 10 21:04:40.610161 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.610140 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n"] Sep 10 21:04:40.695342 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.695303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csgk\" (UniqueName: \"kubernetes.io/projected/b9ebf889-5ba5-433f-8fc8-f3be0623cc58-kube-api-access-2csgk\") pod \"limitador-operator-controller-manager-74495c69f8-lkw8n\" (UID: \"b9ebf889-5ba5-433f-8fc8-f3be0623cc58\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:04:40.796203 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.796171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csgk\" (UniqueName: \"kubernetes.io/projected/b9ebf889-5ba5-433f-8fc8-f3be0623cc58-kube-api-access-2csgk\") pod \"limitador-operator-controller-manager-74495c69f8-lkw8n\" (UID: \"b9ebf889-5ba5-433f-8fc8-f3be0623cc58\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:04:40.817659 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.817625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csgk\" (UniqueName: \"kubernetes.io/projected/b9ebf889-5ba5-433f-8fc8-f3be0623cc58-kube-api-access-2csgk\") pod \"limitador-operator-controller-manager-74495c69f8-lkw8n\" (UID: \"b9ebf889-5ba5-433f-8fc8-f3be0623cc58\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:04:40.913956 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:40.913870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:04:41.179501 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.179468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:41.359768 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.359743 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n"] Sep 10 21:04:41.361848 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:04:41.361817 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ebf889_5ba5_433f_8fc8_f3be0623cc58.slice/crio-8949181100df08b773dc4db6e503ef9bb7b10b6d01c4bb331c10c4244467018e WatchSource:0}: Error finding container 8949181100df08b773dc4db6e503ef9bb7b10b6d01c4bb331c10c4244467018e: Status 404 returned error can't find the container with id 8949181100df08b773dc4db6e503ef9bb7b10b6d01c4bb331c10c4244467018e Sep 10 21:04:41.577228 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.577196 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:41.801778 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.801740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" event={"ID":"b9ebf889-5ba5-433f-8fc8-f3be0623cc58","Type":"ContainerStarted","Data":"8949181100df08b773dc4db6e503ef9bb7b10b6d01c4bb331c10c4244467018e"} Sep 10 21:04:41.803404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.803373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" event={"ID":"f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018","Type":"ContainerStarted","Data":"0a86a169fbc831aa268e455da43b5a92f7b93254010a21fa3daa81ae43fdbfc8"} Sep 10 21:04:41.803552 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.803431 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" Sep 10 21:04:41.843218 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.843131 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" podStartSLOduration=1.4175327690000001 podStartE2EDuration="3.843117088s" totalImagesPullingTime="2.425584319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:41.840213067 +0000 UTC m=+467.658787213" watchObservedRunningTime="2026-09-10 21:04:41.843117088 +0000 UTC m=+467.661691209" Sep 10 21:04:41.970163 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:41.970127 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:42.252040 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:42.251954 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:42.366436 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:42.366397 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:42.585483 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:42.585441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:42.809440 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:42.809409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" event={"ID":"b9ebf889-5ba5-433f-8fc8-f3be0623cc58","Type":"ContainerStarted","Data":"911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0"} Sep 10 21:04:42.809788 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:42.809601 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:04:42.835316 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:42.835245 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" podStartSLOduration=1.490428671 podStartE2EDuration="2.835226217s" totalImagesPullingTime="1.344797546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:42.834558548 +0000 UTC m=+468.653132670" watchObservedRunningTime="2026-09-10 21:04:42.835226217 +0000 UTC m=+468.653800352" Sep 10 21:04:46.368520 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:46.368486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:46.468944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:46.468913 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:46.988420 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:46.988379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:47.162596 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:47.162557 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:49.486952 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:49.486919 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:49.589200 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:49.589145 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:52.455308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:52.455264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:52.782382 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:52.782346 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:52.811711 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:52.811682 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lh6qp" Sep 10 21:04:53.820186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:53.820159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:04:53.890348 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:53.890313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:54.028231 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:54.028199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:58.662351 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:58.662314 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:58.780499 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:58.780465 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:04:58.882224 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:04:58.882190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:04.126459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.126425 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd"] Sep 10 21:05:04.129912 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.129895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.132422 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.132400 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vlnlh\"" Sep 10 21:05:04.144027 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.144003 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n"] Sep 10 21:05:04.144202 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.144184 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" containerName="manager" containerID="cri-o://911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0" gracePeriod=2 Sep 10 21:05:04.160994 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.160971 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd"] Sep 10 21:05:04.165781 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.165758 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n"] Sep 10 21:05:04.206120 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.206081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8zc\" (UniqueName: \"kubernetes.io/projected/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-kube-api-access-dx8zc\") pod \"kuadrant-operator-controller-manager-5f685c496f-st8bd\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.206265 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.206146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-st8bd\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.223175 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:05:04.223140 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-lkw8n\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:05:04.306891 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.306856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8zc\" (UniqueName: \"kubernetes.io/projected/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-kube-api-access-dx8zc\") pod \"kuadrant-operator-controller-manager-5f685c496f-st8bd\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.307054 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.306901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-st8bd\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.307199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.307183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-st8bd\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.320918 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.320888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8zc\" (UniqueName: \"kubernetes.io/projected/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-kube-api-access-dx8zc\") pod \"kuadrant-operator-controller-manager-5f685c496f-st8bd\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.384364 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.384311 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:05:04.388340 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:05:04.388315 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-lkw8n\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:05:04.407670 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.407646 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2csgk\" (UniqueName: \"kubernetes.io/projected/b9ebf889-5ba5-433f-8fc8-f3be0623cc58-kube-api-access-2csgk\") pod \"b9ebf889-5ba5-433f-8fc8-f3be0623cc58\" (UID: \"b9ebf889-5ba5-433f-8fc8-f3be0623cc58\") " Sep 10 21:05:04.409951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.409922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ebf889-5ba5-433f-8fc8-f3be0623cc58-kube-api-access-2csgk" (OuterVolumeSpecName: "kube-api-access-2csgk") pod "b9ebf889-5ba5-433f-8fc8-f3be0623cc58" (UID: "b9ebf889-5ba5-433f-8fc8-f3be0623cc58"). InnerVolumeSpecName "kube-api-access-2csgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:05:04.441454 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.441417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:04.508717 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.508687 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2csgk\" (UniqueName: \"kubernetes.io/projected/b9ebf889-5ba5-433f-8fc8-f3be0623cc58-kube-api-access-2csgk\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:05:04.649509 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:05:04.649428 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-lkw8n\" is forbidden: User \"system:node:ip-10-0-132-255.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-255.ec2.internal' and this object" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:05:04.649969 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.649949 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" path="/var/lib/kubelet/pods/b9ebf889-5ba5-433f-8fc8-f3be0623cc58/volumes" Sep 10 21:05:04.789308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.789279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd"] Sep 10 21:05:04.793007 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:05:04.792983 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd2aa4d_c1cb_44a7_82ba_ea5bcb8c1118.slice/crio-622a326a4bd5f69f66bac7c049a6066a05a939b83b514abae4fe0615aeab3627 WatchSource:0}: Error finding container 622a326a4bd5f69f66bac7c049a6066a05a939b83b514abae4fe0615aeab3627: Status 404 returned error can't find the container with id 622a326a4bd5f69f66bac7c049a6066a05a939b83b514abae4fe0615aeab3627 Sep 10 21:05:04.921547 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.921445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" event={"ID":"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118","Type":"ContainerStarted","Data":"622a326a4bd5f69f66bac7c049a6066a05a939b83b514abae4fe0615aeab3627"} Sep 10 21:05:04.922713 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.922677 2574 generic.go:356] "Generic (PLEG): container finished" podID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" containerID="911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0" exitCode=0 Sep 10 21:05:04.922841 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.922719 2574 scope.go:117] "RemoveContainer" containerID="911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0" Sep 10 21:05:04.922841 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.922734 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lkw8n" Sep 10 21:05:04.931324 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.931305 2574 scope.go:117] "RemoveContainer" containerID="911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0" Sep 10 21:05:04.931589 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:05:04.931569 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0\": container with ID starting with 911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0 not found: ID does not exist" containerID="911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0" Sep 10 21:05:04.931638 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:04.931598 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0"} err="failed to get container status \"911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0\": rpc error: code = NotFound desc = could not find container \"911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0\": container with ID starting with 911e00cd0802a025f471c3d64aac7d9ab69e1861eae5e83045f2afa38eaed0e0 not found: ID does not exist" Sep 10 21:05:06.110317 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:06.110278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:06.389628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:06.389498 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:06.699843 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:06.699731 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:08.946377 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:08.946339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" event={"ID":"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118","Type":"ContainerStarted","Data":"60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a"} Sep 10 21:05:08.946871 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:08.946461 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:09.008327 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:09.008274 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" podStartSLOduration=1.447493152 podStartE2EDuration="5.008257353s" totalImagesPullingTime="3.560764201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:09.007952994 +0000 UTC m=+494.826527116" watchObservedRunningTime="2026-09-10 21:05:09.008257353 +0000 UTC m=+494.826831473" Sep 10 21:05:09.679556 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:09.679523 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:09.867672 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:09.867640 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:15.282762 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:15.282720 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:15.659779 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:15.659683 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:16.748471 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:16.748438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:19.817663 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:19.817632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:19.940684 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:19.940650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:19.953431 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:19.953406 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:20.062862 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:20.062802 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:33.165357 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.165320 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd"] Sep 10 21:05:33.165829 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.165522 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" podUID="6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" containerName="manager" containerID="cri-o://60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a" gracePeriod=10 Sep 10 21:05:33.407924 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.407901 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:33.571734 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.571701 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-extensions-socket-volume\") pod \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " Sep 10 21:05:33.571940 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.571743 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx8zc\" (UniqueName: \"kubernetes.io/projected/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-kube-api-access-dx8zc\") pod \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\" (UID: \"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118\") " Sep 10 21:05:33.572288 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.572266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" (UID: "6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:05:33.574038 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.574014 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-kube-api-access-dx8zc" (OuterVolumeSpecName: "kube-api-access-dx8zc") pod "6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" (UID: "6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118"). InnerVolumeSpecName "kube-api-access-dx8zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:05:33.672752 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.672717 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-extensions-socket-volume\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:05:33.672752 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:33.672749 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8zc\" (UniqueName: \"kubernetes.io/projected/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118-kube-api-access-dx8zc\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:05:34.064616 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.064577 2574 generic.go:356] "Generic (PLEG): container finished" podID="6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" containerID="60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a" exitCode=0 Sep 10 21:05:34.064616 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.064621 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" Sep 10 21:05:34.064829 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.064660 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" event={"ID":"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118","Type":"ContainerDied","Data":"60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a"} Sep 10 21:05:34.064829 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.064717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd" event={"ID":"6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118","Type":"ContainerDied","Data":"622a326a4bd5f69f66bac7c049a6066a05a939b83b514abae4fe0615aeab3627"} Sep 10 21:05:34.064829 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.064739 2574 scope.go:117] "RemoveContainer" containerID="60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a" Sep 10 21:05:34.073592 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.073577 2574 scope.go:117] "RemoveContainer" containerID="60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a" Sep 10 21:05:34.073866 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:05:34.073849 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a\": container with ID starting with 60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a not found: ID does not exist" containerID="60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a" Sep 10 21:05:34.073923 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.073875 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a"} err="failed to get container status \"60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a\": rpc error: code = NotFound desc = could not find container \"60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a\": container with ID starting with 60f45cc7b26def40889e57212daff6b5dbbe86cd4e04d86905a337bc1af6277a not found: ID does not exist" Sep 10 21:05:34.093654 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.093630 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd"] Sep 10 21:05:34.096753 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.096734 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-st8bd"] Sep 10 21:05:34.649830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:34.649781 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" path="/var/lib/kubelet/pods/6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118/volumes" Sep 10 21:05:36.428012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.427981 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:36.506943 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.506896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:36.666725 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.666694 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f"] Sep 10 21:05:36.667055 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.667042 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" containerName="manager" Sep 10 21:05:36.667103 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.667056 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" containerName="manager" Sep 10 21:05:36.667103 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.667074 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" containerName="manager" Sep 10 21:05:36.667103 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.667079 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" containerName="manager" Sep 10 21:05:36.667212 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.667135 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ebf889-5ba5-433f-8fc8-f3be0623cc58" containerName="manager" Sep 10 21:05:36.667212 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.667144 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd2aa4d-c1cb-44a7-82ba-ea5bcb8c1118" containerName="manager" Sep 10 21:05:36.671205 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.671187 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.674630 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.674609 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-twwx9\"" Sep 10 21:05:36.708652 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.708626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f"] Sep 10 21:05:36.715590 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.715563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:36.799120 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799257 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a2ca1c-58c8-445f-a151-3875cc3b4421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799257 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799257 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799395 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6md\" (UniqueName: \"kubernetes.io/projected/58a2ca1c-58c8-445f-a151-3875cc3b4421-kube-api-access-bn6md\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799395 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799470 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799470 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.799470 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.799437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.898717 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.898660 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:36.900739 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.900717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6md\" (UniqueName: \"kubernetes.io/projected/58a2ca1c-58c8-445f-a151-3875cc3b4421-kube-api-access-bn6md\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.900830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.900748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.900830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.900802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.900950 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.900836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.900950 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.900851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.900950 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.900871 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901090 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a2ca1c-58c8-445f-a151-3875cc3b4421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901141 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901181 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901231 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901285 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901343 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901432 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.901711 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.901688 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a2ca1c-58c8-445f-a151-3875cc3b4421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.903383 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.903357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.903648 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.903631 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.909323 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.909301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6md\" (UniqueName: \"kubernetes.io/projected/58a2ca1c-58c8-445f-a151-3875cc3b4421-kube-api-access-bn6md\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.909406 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.909326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a2ca1c-58c8-445f-a151-3875cc3b4421-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lf76f\" (UID: \"58a2ca1c-58c8-445f-a151-3875cc3b4421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:36.983446 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:36.983374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:37.270606 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:37.270583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f"] Sep 10 21:05:37.272565 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:05:37.272539 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a2ca1c_58c8_445f_a151_3875cc3b4421.slice/crio-86a26f588bc1c8fdf8ba095e00786616cbc864ac49cfa93e6af8455ee234cac3 WatchSource:0}: Error finding container 86a26f588bc1c8fdf8ba095e00786616cbc864ac49cfa93e6af8455ee234cac3: Status 404 returned error can't find the container with id 86a26f588bc1c8fdf8ba095e00786616cbc864ac49cfa93e6af8455ee234cac3 Sep 10 21:05:37.275335 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:37.275302 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:05:37.275433 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:37.275379 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:05:37.275433 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:37.275426 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:05:38.086336 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:38.086303 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" event={"ID":"58a2ca1c-58c8-445f-a151-3875cc3b4421","Type":"ContainerStarted","Data":"5c70e27c67e7fdc9b224368156ff17b91dfc8474400ea1d488915c823a1ee1a5"} Sep 10 21:05:38.086757 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:38.086353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" event={"ID":"58a2ca1c-58c8-445f-a151-3875cc3b4421","Type":"ContainerStarted","Data":"86a26f588bc1c8fdf8ba095e00786616cbc864ac49cfa93e6af8455ee234cac3"} Sep 10 21:05:38.144030 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:38.143974 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" podStartSLOduration=2.143956901 podStartE2EDuration="2.143956901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:38.143070093 +0000 UTC m=+523.961644253" watchObservedRunningTime="2026-09-10 21:05:38.143956901 +0000 UTC m=+523.962531021" Sep 10 21:05:38.984451 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:38.984421 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:38.988971 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:38.988950 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:39.091135 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:39.091099 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:39.092071 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:39.092049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:05:49.962500 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:49.962423 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vnqc4"] Sep 10 21:05:49.965880 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:49.965855 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:49.966101 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:49.966064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:49.968334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:49.968317 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6w8c5\"" Sep 10 21:05:49.968682 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:49.968664 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 21:05:49.975601 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:49.975578 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vnqc4"] Sep 10 21:05:50.018540 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.018513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82shj\" (UniqueName: \"kubernetes.io/projected/40cb83d1-db10-40e7-92a7-eb1744def72b-kube-api-access-82shj\") pod \"limitador-limitador-8588749977-vnqc4\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:50.018682 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.018586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40cb83d1-db10-40e7-92a7-eb1744def72b-config-file\") pod \"limitador-limitador-8588749977-vnqc4\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:50.076043 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.076016 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vnqc4"] Sep 10 21:05:50.101443 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.101417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:50.119665 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.119642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82shj\" (UniqueName: \"kubernetes.io/projected/40cb83d1-db10-40e7-92a7-eb1744def72b-kube-api-access-82shj\") pod \"limitador-limitador-8588749977-vnqc4\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:50.119780 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.119683 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40cb83d1-db10-40e7-92a7-eb1744def72b-config-file\") pod \"limitador-limitador-8588749977-vnqc4\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:50.120242 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.120225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40cb83d1-db10-40e7-92a7-eb1744def72b-config-file\") pod \"limitador-limitador-8588749977-vnqc4\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:50.132562 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.132542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82shj\" (UniqueName: \"kubernetes.io/projected/40cb83d1-db10-40e7-92a7-eb1744def72b-kube-api-access-82shj\") pod \"limitador-limitador-8588749977-vnqc4\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:50.243211 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.243135 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:50.277055 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.277027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:50.550930 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.550909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vnqc4"] Sep 10 21:05:50.553193 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:05:50.553167 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cb83d1_db10_40e7_92a7_eb1744def72b.slice/crio-5f1c008b17619e103503ea4d3d2a522fd14bbd568212dfdf373e50326e51091f WatchSource:0}: Error finding container 5f1c008b17619e103503ea4d3d2a522fd14bbd568212dfdf373e50326e51091f: Status 404 returned error can't find the container with id 5f1c008b17619e103503ea4d3d2a522fd14bbd568212dfdf373e50326e51091f Sep 10 21:05:50.826919 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.826824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:50.917073 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.917034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:05:50.918273 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.918250 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qfkc9"] Sep 10 21:05:50.922731 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.922700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qfkc9" Sep 10 21:05:50.925702 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.925680 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cqhw2\"" Sep 10 21:05:50.933145 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:50.933120 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qfkc9"] Sep 10 21:05:51.027258 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:51.027229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95bz\" (UniqueName: \"kubernetes.io/projected/cbb01920-7f0a-4d50-a2ea-998e010eb36d-kube-api-access-x95bz\") pod \"authorino-59c974bd48-qfkc9\" (UID: \"cbb01920-7f0a-4d50-a2ea-998e010eb36d\") " pod="kuadrant-system/authorino-59c974bd48-qfkc9" Sep 10 21:05:51.128370 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:51.128297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95bz\" (UniqueName: \"kubernetes.io/projected/cbb01920-7f0a-4d50-a2ea-998e010eb36d-kube-api-access-x95bz\") pod \"authorino-59c974bd48-qfkc9\" (UID: \"cbb01920-7f0a-4d50-a2ea-998e010eb36d\") " pod="kuadrant-system/authorino-59c974bd48-qfkc9" Sep 10 21:05:51.139160 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:51.139138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95bz\" (UniqueName: \"kubernetes.io/projected/cbb01920-7f0a-4d50-a2ea-998e010eb36d-kube-api-access-x95bz\") pod \"authorino-59c974bd48-qfkc9\" (UID: \"cbb01920-7f0a-4d50-a2ea-998e010eb36d\") " pod="kuadrant-system/authorino-59c974bd48-qfkc9" Sep 10 21:05:51.149316 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:51.149292 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" event={"ID":"40cb83d1-db10-40e7-92a7-eb1744def72b","Type":"ContainerStarted","Data":"5f1c008b17619e103503ea4d3d2a522fd14bbd568212dfdf373e50326e51091f"} Sep 10 21:05:51.232038 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:51.232015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qfkc9" Sep 10 21:05:51.505007 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:51.504976 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qfkc9"] Sep 10 21:05:51.641731 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:05:51.641702 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbb01920_7f0a_4d50_a2ea_998e010eb36d.slice/crio-0fdfda28d355834dd2a37bddd9d7b77d2e6e1b58b2e8a76db8a27af9bddba610 WatchSource:0}: Error finding container 0fdfda28d355834dd2a37bddd9d7b77d2e6e1b58b2e8a76db8a27af9bddba610: Status 404 returned error can't find the container with id 0fdfda28d355834dd2a37bddd9d7b77d2e6e1b58b2e8a76db8a27af9bddba610 Sep 10 21:05:52.157417 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:52.157383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" event={"ID":"40cb83d1-db10-40e7-92a7-eb1744def72b","Type":"ContainerStarted","Data":"9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b"} Sep 10 21:05:52.157900 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:52.157586 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:05:52.158949 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:52.158924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qfkc9" event={"ID":"cbb01920-7f0a-4d50-a2ea-998e010eb36d","Type":"ContainerStarted","Data":"0fdfda28d355834dd2a37bddd9d7b77d2e6e1b58b2e8a76db8a27af9bddba610"} Sep 10 21:05:52.180789 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:52.180741 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" podStartSLOduration=2.030541516 podStartE2EDuration="3.180727897s" totalImagesPullingTime="1.150186381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:52.179343939 +0000 UTC m=+537.997918060" watchObservedRunningTime="2026-09-10 21:05:52.180727897 +0000 UTC m=+537.999302018" Sep 10 21:05:53.166638 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:53.166599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qfkc9" event={"ID":"cbb01920-7f0a-4d50-a2ea-998e010eb36d","Type":"ContainerStarted","Data":"6dd222a066d9dbc419c82e805a76bf1da1713d48d2c6bd1bf2f6ddf130cdc526"} Sep 10 21:05:53.188520 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:05:53.188110 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qfkc9" podStartSLOduration=1.784452023 podStartE2EDuration="3.188086679s" totalImagesPullingTime="1.403634656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:53.187037347 +0000 UTC m=+539.005611468" watchObservedRunningTime="2026-09-10 21:05:53.188086679 +0000 UTC m=+539.006660808" Sep 10 21:06:03.168302 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:03.168273 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:06:06.124456 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.124417 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vnqc4"] Sep 10 21:06:06.124998 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.124653 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" podUID="40cb83d1-db10-40e7-92a7-eb1744def72b" containerName="limitador" containerID="cri-o://9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b" gracePeriod=30 Sep 10 21:06:06.671555 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.671532 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:06:06.765110 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.765033 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82shj\" (UniqueName: \"kubernetes.io/projected/40cb83d1-db10-40e7-92a7-eb1744def72b-kube-api-access-82shj\") pod \"40cb83d1-db10-40e7-92a7-eb1744def72b\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " Sep 10 21:06:06.765110 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.765080 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40cb83d1-db10-40e7-92a7-eb1744def72b-config-file\") pod \"40cb83d1-db10-40e7-92a7-eb1744def72b\" (UID: \"40cb83d1-db10-40e7-92a7-eb1744def72b\") " Sep 10 21:06:06.765506 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.765478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cb83d1-db10-40e7-92a7-eb1744def72b-config-file" (OuterVolumeSpecName: "config-file") pod "40cb83d1-db10-40e7-92a7-eb1744def72b" (UID: "40cb83d1-db10-40e7-92a7-eb1744def72b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:06:06.767331 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.767302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cb83d1-db10-40e7-92a7-eb1744def72b-kube-api-access-82shj" (OuterVolumeSpecName: "kube-api-access-82shj") pod "40cb83d1-db10-40e7-92a7-eb1744def72b" (UID: "40cb83d1-db10-40e7-92a7-eb1744def72b"). InnerVolumeSpecName "kube-api-access-82shj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:06:06.865901 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.865863 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40cb83d1-db10-40e7-92a7-eb1744def72b-config-file\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:06:06.865901 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:06.865894 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82shj\" (UniqueName: \"kubernetes.io/projected/40cb83d1-db10-40e7-92a7-eb1744def72b-kube-api-access-82shj\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:06:07.236830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.236780 2574 generic.go:356] "Generic (PLEG): container finished" podID="40cb83d1-db10-40e7-92a7-eb1744def72b" containerID="9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b" exitCode=0 Sep 10 21:06:07.237199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.236851 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" Sep 10 21:06:07.237199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.236872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" event={"ID":"40cb83d1-db10-40e7-92a7-eb1744def72b","Type":"ContainerDied","Data":"9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b"} Sep 10 21:06:07.237199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.236933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-vnqc4" event={"ID":"40cb83d1-db10-40e7-92a7-eb1744def72b","Type":"ContainerDied","Data":"5f1c008b17619e103503ea4d3d2a522fd14bbd568212dfdf373e50326e51091f"} Sep 10 21:06:07.237199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.236962 2574 scope.go:117] "RemoveContainer" containerID="9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b" Sep 10 21:06:07.252768 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.252745 2574 scope.go:117] "RemoveContainer" containerID="9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b" Sep 10 21:06:07.253076 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:06:07.253055 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b\": container with ID starting with 9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b not found: ID does not exist" containerID="9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b" Sep 10 21:06:07.253150 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.253088 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b"} err="failed to get container status \"9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b\": rpc error: code = NotFound desc = could not find container \"9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b\": container with ID starting with 9c32cda5fffc11e73abe18d864583e6ff34bbee6d7492dbfe782d876d3bb241b not found: ID does not exist" Sep 10 21:06:07.264492 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.264470 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vnqc4"] Sep 10 21:06:07.270208 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:07.270182 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-vnqc4"] Sep 10 21:06:08.650376 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:08.650339 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40cb83d1-db10-40e7-92a7-eb1744def72b" path="/var/lib/kubelet/pods/40cb83d1-db10-40e7-92a7-eb1744def72b/volumes" Sep 10 21:06:08.707112 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:08.707073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:08.789598 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:08.789564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:10.699198 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:10.699163 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:10.794591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:10.794554 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:11.105450 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.105420 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7"] Sep 10 21:06:11.105830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.105790 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40cb83d1-db10-40e7-92a7-eb1744def72b" containerName="limitador" Sep 10 21:06:11.105830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.105820 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40cb83d1-db10-40e7-92a7-eb1744def72b" containerName="limitador" Sep 10 21:06:11.105969 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.105889 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="40cb83d1-db10-40e7-92a7-eb1744def72b" containerName="limitador" Sep 10 21:06:11.110402 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.110382 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.113039 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.113013 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 21:06:11.113177 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.113076 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 10 21:06:11.113883 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.113860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d86ks\"" Sep 10 21:06:11.113883 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.113873 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 21:06:11.120362 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.120341 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7"] Sep 10 21:06:11.142569 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.142539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:11.200325 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.200294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f06121b9-4285-4b6d-a17e-92862bdbc7bc-data\") pod \"postgres-69fb4594cf-8rnk7\" (UID: \"f06121b9-4285-4b6d-a17e-92862bdbc7bc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.200325 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.200325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56j7p\" (UniqueName: \"kubernetes.io/projected/f06121b9-4285-4b6d-a17e-92862bdbc7bc-kube-api-access-56j7p\") pod \"postgres-69fb4594cf-8rnk7\" (UID: \"f06121b9-4285-4b6d-a17e-92862bdbc7bc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.300853 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.300818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f06121b9-4285-4b6d-a17e-92862bdbc7bc-data\") pod \"postgres-69fb4594cf-8rnk7\" (UID: \"f06121b9-4285-4b6d-a17e-92862bdbc7bc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.300853 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.300854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56j7p\" (UniqueName: \"kubernetes.io/projected/f06121b9-4285-4b6d-a17e-92862bdbc7bc-kube-api-access-56j7p\") pod \"postgres-69fb4594cf-8rnk7\" (UID: \"f06121b9-4285-4b6d-a17e-92862bdbc7bc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.301127 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.301107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f06121b9-4285-4b6d-a17e-92862bdbc7bc-data\") pod \"postgres-69fb4594cf-8rnk7\" (UID: \"f06121b9-4285-4b6d-a17e-92862bdbc7bc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.310126 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.310102 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56j7p\" (UniqueName: \"kubernetes.io/projected/f06121b9-4285-4b6d-a17e-92862bdbc7bc-kube-api-access-56j7p\") pod \"postgres-69fb4594cf-8rnk7\" (UID: \"f06121b9-4285-4b6d-a17e-92862bdbc7bc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.421681 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.421604 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:11.696432 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:11.696327 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7"] Sep 10 21:06:11.698659 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:06:11.698613 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06121b9_4285_4b6d_a17e_92862bdbc7bc.slice/crio-a40bd10fc79622ec7f6be9536ac1ce1ccc6cd44b01ba039583389c156ca2a9c0 WatchSource:0}: Error finding container a40bd10fc79622ec7f6be9536ac1ce1ccc6cd44b01ba039583389c156ca2a9c0: Status 404 returned error can't find the container with id a40bd10fc79622ec7f6be9536ac1ce1ccc6cd44b01ba039583389c156ca2a9c0 Sep 10 21:06:12.265369 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:12.265296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" event={"ID":"f06121b9-4285-4b6d-a17e-92862bdbc7bc","Type":"ContainerStarted","Data":"a40bd10fc79622ec7f6be9536ac1ce1ccc6cd44b01ba039583389c156ca2a9c0"} Sep 10 21:06:17.294552 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:17.294514 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" event={"ID":"f06121b9-4285-4b6d-a17e-92862bdbc7bc","Type":"ContainerStarted","Data":"1df6fc0b3212b841695d6aa012a209b6e52b95d7abf3e818afe1925b1189ed7b"} Sep 10 21:06:17.295008 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:17.294671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:17.313959 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:17.313915 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" podStartSLOduration=1.69601013 podStartE2EDuration="6.313901355s" totalImagesPullingTime="4.617891225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:17.312312527 +0000 UTC m=+563.130886670" watchObservedRunningTime="2026-09-10 21:06:17.313901355 +0000 UTC m=+563.132475522" Sep 10 21:06:23.332596 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:23.332567 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8rnk7" Sep 10 21:06:23.649193 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:23.649109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:23.871927 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:23.871897 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-lmtfh"] Sep 10 21:06:23.880718 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:23.880699 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lmtfh" Sep 10 21:06:23.885156 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:23.885131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-lmtfh"] Sep 10 21:06:23.966075 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:23.965994 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:24.018254 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.018221 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjw5k\" (UniqueName: \"kubernetes.io/projected/3f23412b-5011-48a8-a324-fe7ca6e4abd9-kube-api-access-cjw5k\") pod \"authorino-557f876f69-lmtfh\" (UID: \"3f23412b-5011-48a8-a324-fe7ca6e4abd9\") " pod="kuadrant-system/authorino-557f876f69-lmtfh" Sep 10 21:06:24.101847 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.101820 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lmtfh"] Sep 10 21:06:24.102137 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:06:24.102107 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cjw5k], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-lmtfh" podUID="3f23412b-5011-48a8-a324-fe7ca6e4abd9" Sep 10 21:06:24.119164 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.119139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjw5k\" (UniqueName: \"kubernetes.io/projected/3f23412b-5011-48a8-a324-fe7ca6e4abd9-kube-api-access-cjw5k\") pod \"authorino-557f876f69-lmtfh\" (UID: \"3f23412b-5011-48a8-a324-fe7ca6e4abd9\") " pod="kuadrant-system/authorino-557f876f69-lmtfh" Sep 10 21:06:24.128302 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.128278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjw5k\" (UniqueName: \"kubernetes.io/projected/3f23412b-5011-48a8-a324-fe7ca6e4abd9-kube-api-access-cjw5k\") pod \"authorino-557f876f69-lmtfh\" (UID: \"3f23412b-5011-48a8-a324-fe7ca6e4abd9\") " pod="kuadrant-system/authorino-557f876f69-lmtfh" Sep 10 21:06:24.329871 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.329838 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lmtfh" Sep 10 21:06:24.411457 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.411427 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lmtfh" Sep 10 21:06:24.523152 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.523121 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjw5k\" (UniqueName: \"kubernetes.io/projected/3f23412b-5011-48a8-a324-fe7ca6e4abd9-kube-api-access-cjw5k\") pod \"3f23412b-5011-48a8-a324-fe7ca6e4abd9\" (UID: \"3f23412b-5011-48a8-a324-fe7ca6e4abd9\") " Sep 10 21:06:24.525276 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.525248 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f23412b-5011-48a8-a324-fe7ca6e4abd9-kube-api-access-cjw5k" (OuterVolumeSpecName: "kube-api-access-cjw5k") pod "3f23412b-5011-48a8-a324-fe7ca6e4abd9" (UID: "3f23412b-5011-48a8-a324-fe7ca6e4abd9"). InnerVolumeSpecName "kube-api-access-cjw5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:06:24.623795 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:24.623720 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjw5k\" (UniqueName: \"kubernetes.io/projected/3f23412b-5011-48a8-a324-fe7ca6e4abd9-kube-api-access-cjw5k\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:06:25.217362 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.217328 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qfkc9"] Sep 10 21:06:25.217577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.217542 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qfkc9" podUID="cbb01920-7f0a-4d50-a2ea-998e010eb36d" containerName="authorino" containerID="cri-o://6dd222a066d9dbc419c82e805a76bf1da1713d48d2c6bd1bf2f6ddf130cdc526" gracePeriod=30 Sep 10 21:06:25.335530 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.335500 2574 generic.go:356] "Generic (PLEG): container finished" podID="cbb01920-7f0a-4d50-a2ea-998e010eb36d" containerID="6dd222a066d9dbc419c82e805a76bf1da1713d48d2c6bd1bf2f6ddf130cdc526" exitCode=0 Sep 10 21:06:25.335681 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.335579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qfkc9" event={"ID":"cbb01920-7f0a-4d50-a2ea-998e010eb36d","Type":"ContainerDied","Data":"6dd222a066d9dbc419c82e805a76bf1da1713d48d2c6bd1bf2f6ddf130cdc526"} Sep 10 21:06:25.335681 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.335640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lmtfh" Sep 10 21:06:25.364210 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.364181 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lmtfh"] Sep 10 21:06:25.368218 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.368194 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-lmtfh"] Sep 10 21:06:25.496116 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.496094 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qfkc9" Sep 10 21:06:25.632290 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.632258 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x95bz\" (UniqueName: \"kubernetes.io/projected/cbb01920-7f0a-4d50-a2ea-998e010eb36d-kube-api-access-x95bz\") pod \"cbb01920-7f0a-4d50-a2ea-998e010eb36d\" (UID: \"cbb01920-7f0a-4d50-a2ea-998e010eb36d\") " Sep 10 21:06:25.634452 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.634429 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb01920-7f0a-4d50-a2ea-998e010eb36d-kube-api-access-x95bz" (OuterVolumeSpecName: "kube-api-access-x95bz") pod "cbb01920-7f0a-4d50-a2ea-998e010eb36d" (UID: "cbb01920-7f0a-4d50-a2ea-998e010eb36d"). InnerVolumeSpecName "kube-api-access-x95bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:06:25.734004 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:25.733929 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95bz\" (UniqueName: \"kubernetes.io/projected/cbb01920-7f0a-4d50-a2ea-998e010eb36d-kube-api-access-x95bz\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:06:26.341225 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:26.341199 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qfkc9" Sep 10 21:06:26.341387 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:26.341244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qfkc9" event={"ID":"cbb01920-7f0a-4d50-a2ea-998e010eb36d","Type":"ContainerDied","Data":"0fdfda28d355834dd2a37bddd9d7b77d2e6e1b58b2e8a76db8a27af9bddba610"} Sep 10 21:06:26.341387 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:26.341284 2574 scope.go:117] "RemoveContainer" containerID="6dd222a066d9dbc419c82e805a76bf1da1713d48d2c6bd1bf2f6ddf130cdc526" Sep 10 21:06:26.370524 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:26.370492 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qfkc9"] Sep 10 21:06:26.370787 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:26.370765 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qfkc9"] Sep 10 21:06:26.650099 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:26.650023 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f23412b-5011-48a8-a324-fe7ca6e4abd9" path="/var/lib/kubelet/pods/3f23412b-5011-48a8-a324-fe7ca6e4abd9/volumes" Sep 10 21:06:26.650425 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:26.650229 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb01920-7f0a-4d50-a2ea-998e010eb36d" path="/var/lib/kubelet/pods/cbb01920-7f0a-4d50-a2ea-998e010eb36d/volumes" Sep 10 21:06:29.929665 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.929628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:29.977526 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.977496 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7867c6597c-tt56s"] Sep 10 21:06:29.978278 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.978255 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb01920-7f0a-4d50-a2ea-998e010eb36d" containerName="authorino" Sep 10 21:06:29.978278 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.978279 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb01920-7f0a-4d50-a2ea-998e010eb36d" containerName="authorino" Sep 10 21:06:29.978436 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.978397 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb01920-7f0a-4d50-a2ea-998e010eb36d" containerName="authorino" Sep 10 21:06:29.983076 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.983048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:29.985769 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.985750 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 21:06:29.986147 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.986129 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p25h6\"" Sep 10 21:06:29.986914 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.986824 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 21:06:29.986914 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.986869 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 21:06:29.992404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:29.992381 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7867c6597c-tt56s"] Sep 10 21:06:30.012682 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.012650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:30.072610 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.072574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-webhook-cert\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.072779 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.072623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xbm\" (UniqueName: \"kubernetes.io/projected/ceb96d57-fd39-4660-9ffb-ebe34711e85e-kube-api-access-m8xbm\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.072779 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.072672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.092384 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.092361 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:30.149594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.149563 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-k6nk9"] Sep 10 21:06:30.152061 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.152042 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.164929 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.164908 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-k6nk9"] Sep 10 21:06:30.173399 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.173372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-webhook-cert\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.173514 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.173436 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xbm\" (UniqueName: \"kubernetes.io/projected/ceb96d57-fd39-4660-9ffb-ebe34711e85e-kube-api-access-m8xbm\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.173514 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.173470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmwc\" (UniqueName: \"kubernetes.io/projected/f97205e8-2170-49a5-a7e3-293de14a38d2-kube-api-access-dwmwc\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.173647 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.173530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-webhook-cert\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.173647 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.173585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.173647 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.173631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-metrics-certs\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.173799 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:06:30.173668 2574 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 10 21:06:30.173799 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:06:30.173722 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs podName:ceb96d57-fd39-4660-9ffb-ebe34711e85e nodeName:}" failed. No retries permitted until 2026-09-10 21:06:30.673702091 +0000 UTC m=+576.492276205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs") pod "maas-controller-7867c6597c-tt56s" (UID: "ceb96d57-fd39-4660-9ffb-ebe34711e85e") : secret "maas-controller-metrics-tls" not found Sep 10 21:06:30.176261 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.176240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-webhook-cert\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.183566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.183517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xbm\" (UniqueName: \"kubernetes.io/projected/ceb96d57-fd39-4660-9ffb-ebe34711e85e-kube-api-access-m8xbm\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.268855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.268799 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:30.274940 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.274916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmwc\" (UniqueName: \"kubernetes.io/projected/f97205e8-2170-49a5-a7e3-293de14a38d2-kube-api-access-dwmwc\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.275076 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.274980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-webhook-cert\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.275076 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.275055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-metrics-certs\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.277591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.277571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-webhook-cert\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.277683 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.277600 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-metrics-certs\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.283798 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.283772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmwc\" (UniqueName: \"kubernetes.io/projected/f97205e8-2170-49a5-a7e3-293de14a38d2-kube-api-access-dwmwc\") pod \"maas-controller-77cdb86f9b-k6nk9\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.462470 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.462386 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:30.679105 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.679068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.681594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.681565 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs\") pod \"maas-controller-7867c6597c-tt56s\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:30.733174 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.733149 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-k6nk9"] Sep 10 21:06:30.734724 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:06:30.734703 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97205e8_2170_49a5_a7e3_293de14a38d2.slice/crio-792b2b945b1b2aa4c6889d2c4a125c7364a3b3898583a0faa17bcc925370d51b WatchSource:0}: Error finding container 792b2b945b1b2aa4c6889d2c4a125c7364a3b3898583a0faa17bcc925370d51b: Status 404 returned error can't find the container with id 792b2b945b1b2aa4c6889d2c4a125c7364a3b3898583a0faa17bcc925370d51b Sep 10 21:06:30.894738 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:30.894707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:31.168822 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:31.168784 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7867c6597c-tt56s"] Sep 10 21:06:31.170452 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:06:31.170430 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb96d57_fd39_4660_9ffb_ebe34711e85e.slice/crio-d6bee0512aa1684599123ac519ff5a2721730272e357d7e228c23a8f1e986f02 WatchSource:0}: Error finding container d6bee0512aa1684599123ac519ff5a2721730272e357d7e228c23a8f1e986f02: Status 404 returned error can't find the container with id d6bee0512aa1684599123ac519ff5a2721730272e357d7e228c23a8f1e986f02 Sep 10 21:06:31.377559 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:31.377517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" event={"ID":"f97205e8-2170-49a5-a7e3-293de14a38d2","Type":"ContainerStarted","Data":"792b2b945b1b2aa4c6889d2c4a125c7364a3b3898583a0faa17bcc925370d51b"} Sep 10 21:06:31.378978 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:31.378951 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867c6597c-tt56s" event={"ID":"ceb96d57-fd39-4660-9ffb-ebe34711e85e","Type":"ContainerStarted","Data":"d6bee0512aa1684599123ac519ff5a2721730272e357d7e228c23a8f1e986f02"} Sep 10 21:06:33.392772 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.392730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" event={"ID":"f97205e8-2170-49a5-a7e3-293de14a38d2","Type":"ContainerStarted","Data":"e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b"} Sep 10 21:06:33.393232 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.392880 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:33.395366 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.395334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867c6597c-tt56s" event={"ID":"ceb96d57-fd39-4660-9ffb-ebe34711e85e","Type":"ContainerStarted","Data":"19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e"} Sep 10 21:06:33.395509 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.395490 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:33.434406 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.434354 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7867c6597c-tt56s" podStartSLOduration=2.292373934 podStartE2EDuration="4.434339111s" totalImagesPullingTime="2.141965177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:33.432037931 +0000 UTC m=+579.250612075" watchObservedRunningTime="2026-09-10 21:06:33.434339111 +0000 UTC m=+579.252913233" Sep 10 21:06:33.434561 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.434442 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" podStartSLOduration=0.866354658 podStartE2EDuration="3.434438057s" totalImagesPullingTime="2.568083399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:33.412765384 +0000 UTC m=+579.231339505" watchObservedRunningTime="2026-09-10 21:06:33.434438057 +0000 UTC m=+579.253012178" Sep 10 21:06:33.516774 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.516740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:33.616589 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:33.616496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:06:44.402058 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.402026 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:44.402530 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.402204 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:06:44.468502 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.468468 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7867c6597c-tt56s"] Sep 10 21:06:44.469172 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.469138 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7867c6597c-tt56s" podUID="ceb96d57-fd39-4660-9ffb-ebe34711e85e" containerName="manager" containerID="cri-o://19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e" gracePeriod=30 Sep 10 21:06:44.718670 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.718651 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:44.806411 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.806388 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-webhook-cert\") pod \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " Sep 10 21:06:44.806584 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.806443 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xbm\" (UniqueName: \"kubernetes.io/projected/ceb96d57-fd39-4660-9ffb-ebe34711e85e-kube-api-access-m8xbm\") pod \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " Sep 10 21:06:44.806584 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.806465 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs\") pod \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\" (UID: \"ceb96d57-fd39-4660-9ffb-ebe34711e85e\") " Sep 10 21:06:44.808771 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.808745 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ceb96d57-fd39-4660-9ffb-ebe34711e85e" (UID: "ceb96d57-fd39-4660-9ffb-ebe34711e85e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:06:44.808771 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.808763 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ceb96d57-fd39-4660-9ffb-ebe34711e85e" (UID: "ceb96d57-fd39-4660-9ffb-ebe34711e85e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:06:44.808978 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.808766 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb96d57-fd39-4660-9ffb-ebe34711e85e-kube-api-access-m8xbm" (OuterVolumeSpecName: "kube-api-access-m8xbm") pod "ceb96d57-fd39-4660-9ffb-ebe34711e85e" (UID: "ceb96d57-fd39-4660-9ffb-ebe34711e85e"). InnerVolumeSpecName "kube-api-access-m8xbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:06:44.907580 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.907553 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xbm\" (UniqueName: \"kubernetes.io/projected/ceb96d57-fd39-4660-9ffb-ebe34711e85e-kube-api-access-m8xbm\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:06:44.907580 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.907578 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:06:44.907759 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:44.907586 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ceb96d57-fd39-4660-9ffb-ebe34711e85e-webhook-cert\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:06:45.454556 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.454523 2574 generic.go:356] "Generic (PLEG): container finished" podID="ceb96d57-fd39-4660-9ffb-ebe34711e85e" containerID="19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e" exitCode=0 Sep 10 21:06:45.454972 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.454571 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7867c6597c-tt56s" Sep 10 21:06:45.454972 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.454576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867c6597c-tt56s" event={"ID":"ceb96d57-fd39-4660-9ffb-ebe34711e85e","Type":"ContainerDied","Data":"19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e"} Sep 10 21:06:45.454972 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.454627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7867c6597c-tt56s" event={"ID":"ceb96d57-fd39-4660-9ffb-ebe34711e85e","Type":"ContainerDied","Data":"d6bee0512aa1684599123ac519ff5a2721730272e357d7e228c23a8f1e986f02"} Sep 10 21:06:45.454972 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.454649 2574 scope.go:117] "RemoveContainer" containerID="19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e" Sep 10 21:06:45.463615 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.463600 2574 scope.go:117] "RemoveContainer" containerID="19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e" Sep 10 21:06:45.463928 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:06:45.463909 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e\": container with ID starting with 19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e not found: ID does not exist" containerID="19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e" Sep 10 21:06:45.464013 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.463935 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e"} err="failed to get container status \"19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e\": rpc error: code = NotFound desc = could not find container \"19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e\": container with ID starting with 19c5bbc9523b8f5ad8c8cbc1baf1b3488523ba4eb6876cbaa6e677430857506e not found: ID does not exist" Sep 10 21:06:45.476757 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.476732 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7867c6597c-tt56s"] Sep 10 21:06:45.478434 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:45.478416 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7867c6597c-tt56s"] Sep 10 21:06:46.650497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:06:46.650452 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb96d57-fd39-4660-9ffb-ebe34711e85e" path="/var/lib/kubelet/pods/ceb96d57-fd39-4660-9ffb-ebe34711e85e/volumes" Sep 10 21:07:06.169605 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:06.169572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:06.288710 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:06.288674 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:07.662254 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.662226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:07.797517 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.797482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:07.984252 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.984162 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq"] Sep 10 21:07:07.984626 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.984608 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb96d57-fd39-4660-9ffb-ebe34711e85e" containerName="manager" Sep 10 21:07:07.984715 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.984630 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb96d57-fd39-4660-9ffb-ebe34711e85e" containerName="manager" Sep 10 21:07:07.984774 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.984730 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb96d57-fd39-4660-9ffb-ebe34711e85e" containerName="manager" Sep 10 21:07:07.986876 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.986847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:07.994993 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.994971 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 21:07:07.997016 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.996998 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9ckhj\"" Sep 10 21:07:07.998098 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:07.998078 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 21:07:08.038193 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.038165 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq"] Sep 10 21:07:08.110255 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.110227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dacc6d-1984-42c4-9274-5a1fbf64d090-openshift-service-ca\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.110439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.110263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-maas-api-tls\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.110439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.110377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wwm\" (UniqueName: \"kubernetes.io/projected/a3dacc6d-1984-42c4-9274-5a1fbf64d090-kube-api-access-v8wwm\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.110551 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.110442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-metrics-certs\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.211270 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.211236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-metrics-certs\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.211439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.211287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dacc6d-1984-42c4-9274-5a1fbf64d090-openshift-service-ca\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.211439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.211319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-maas-api-tls\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.211439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.211427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wwm\" (UniqueName: \"kubernetes.io/projected/a3dacc6d-1984-42c4-9274-5a1fbf64d090-kube-api-access-v8wwm\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.212046 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.212024 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dacc6d-1984-42c4-9274-5a1fbf64d090-openshift-service-ca\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.213893 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.213867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-metrics-certs\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.213978 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.213912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-maas-api-tls\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.216168 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.216142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:08.238465 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.238408 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wwm\" (UniqueName: \"kubernetes.io/projected/a3dacc6d-1984-42c4-9274-5a1fbf64d090-kube-api-access-v8wwm\") pod \"maas-api-7b78667ccb-dtkzq\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.296188 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.296163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:08.612183 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.612160 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq"] Sep 10 21:07:08.615019 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:07:08.614988 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dacc6d_1984_42c4_9274_5a1fbf64d090.slice/crio-06b3f75770fe22119b151e4c842733243990b65db64a93e1bedd5a5d6166f9ed WatchSource:0}: Error finding container 06b3f75770fe22119b151e4c842733243990b65db64a93e1bedd5a5d6166f9ed: Status 404 returned error can't find the container with id 06b3f75770fe22119b151e4c842733243990b65db64a93e1bedd5a5d6166f9ed Sep 10 21:07:08.971544 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:08.971475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:09.199369 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.199333 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj"] Sep 10 21:07:09.202457 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.202439 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.208218 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.208196 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rdmqj\"" Sep 10 21:07:09.208480 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.208463 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 21:07:09.218722 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.218700 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj"] Sep 10 21:07:09.321704 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.321654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prpt\" (UniqueName: \"kubernetes.io/projected/9c873f35-55af-4bd2-af81-f34a6d48fc23-kube-api-access-9prpt\") pod \"payload-pre-processing-59fbc47bf8-nbkbj\" (UID: \"9c873f35-55af-4bd2-af81-f34a6d48fc23\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.321916 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.321720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c873f35-55af-4bd2-af81-f34a6d48fc23-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-nbkbj\" (UID: \"9c873f35-55af-4bd2-af81-f34a6d48fc23\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.423305 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.423262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prpt\" (UniqueName: \"kubernetes.io/projected/9c873f35-55af-4bd2-af81-f34a6d48fc23-kube-api-access-9prpt\") pod \"payload-pre-processing-59fbc47bf8-nbkbj\" (UID: \"9c873f35-55af-4bd2-af81-f34a6d48fc23\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.423466 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.423354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c873f35-55af-4bd2-af81-f34a6d48fc23-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-nbkbj\" (UID: \"9c873f35-55af-4bd2-af81-f34a6d48fc23\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.424411 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.424339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c873f35-55af-4bd2-af81-f34a6d48fc23-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-nbkbj\" (UID: \"9c873f35-55af-4bd2-af81-f34a6d48fc23\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.436027 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.435966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prpt\" (UniqueName: \"kubernetes.io/projected/9c873f35-55af-4bd2-af81-f34a6d48fc23-kube-api-access-9prpt\") pod \"payload-pre-processing-59fbc47bf8-nbkbj\" (UID: \"9c873f35-55af-4bd2-af81-f34a6d48fc23\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.470360 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.467935 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lx7lp"] Sep 10 21:07:09.473830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.473408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.512414 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.512386 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:09.558873 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.555301 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lx7lp"] Sep 10 21:07:09.578962 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.578878 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" event={"ID":"a3dacc6d-1984-42c4-9274-5a1fbf64d090","Type":"ContainerStarted","Data":"06b3f75770fe22119b151e4c842733243990b65db64a93e1bedd5a5d6166f9ed"} Sep 10 21:07:09.625208 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.625180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7cr\" (UniqueName: \"kubernetes.io/projected/ed07e49a-5e29-4489-93fd-6f96bba473db-kube-api-access-np7cr\") pod \"payload-processing-5b54b7fbb7-lx7lp\" (UID: \"ed07e49a-5e29-4489-93fd-6f96bba473db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.625351 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.625250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07e49a-5e29-4489-93fd-6f96bba473db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lx7lp\" (UID: \"ed07e49a-5e29-4489-93fd-6f96bba473db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.726643 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.726609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07e49a-5e29-4489-93fd-6f96bba473db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lx7lp\" (UID: \"ed07e49a-5e29-4489-93fd-6f96bba473db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.726871 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.726732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7cr\" (UniqueName: \"kubernetes.io/projected/ed07e49a-5e29-4489-93fd-6f96bba473db-kube-api-access-np7cr\") pod \"payload-processing-5b54b7fbb7-lx7lp\" (UID: \"ed07e49a-5e29-4489-93fd-6f96bba473db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.727621 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.727597 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07e49a-5e29-4489-93fd-6f96bba473db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-lx7lp\" (UID: \"ed07e49a-5e29-4489-93fd-6f96bba473db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.758136 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.757863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7cr\" (UniqueName: \"kubernetes.io/projected/ed07e49a-5e29-4489-93fd-6f96bba473db-kube-api-access-np7cr\") pod \"payload-processing-5b54b7fbb7-lx7lp\" (UID: \"ed07e49a-5e29-4489-93fd-6f96bba473db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.785787 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.785755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:09.841057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:09.840149 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj"] Sep 10 21:07:10.096205 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:10.096181 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-lx7lp"] Sep 10 21:07:10.470266 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:10.470167 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:10.587158 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:10.587126 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" event={"ID":"9c873f35-55af-4bd2-af81-f34a6d48fc23","Type":"ContainerStarted","Data":"b170a14f68b19405e386a55e7a488322a67876f0c8bd021f50c10cabb165bed9"} Sep 10 21:07:10.674839 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:07:10.674780 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded07e49a_5e29_4489_93fd_6f96bba473db.slice/crio-ab2400d0a64b001b0be478bf64df44eef741d40ad21cb345c6e7d12d603274fd WatchSource:0}: Error finding container ab2400d0a64b001b0be478bf64df44eef741d40ad21cb345c6e7d12d603274fd: Status 404 returned error can't find the container with id ab2400d0a64b001b0be478bf64df44eef741d40ad21cb345c6e7d12d603274fd Sep 10 21:07:11.597276 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:11.597232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" event={"ID":"a3dacc6d-1984-42c4-9274-5a1fbf64d090","Type":"ContainerStarted","Data":"7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d"} Sep 10 21:07:11.597852 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:11.597314 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:11.599952 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:11.599924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" event={"ID":"ed07e49a-5e29-4489-93fd-6f96bba473db","Type":"ContainerStarted","Data":"ab2400d0a64b001b0be478bf64df44eef741d40ad21cb345c6e7d12d603274fd"} Sep 10 21:07:11.626766 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:11.625389 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" podStartSLOduration=2.513627545 podStartE2EDuration="4.62537003s" totalImagesPullingTime="2.111742485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:11.625050973 +0000 UTC m=+617.443625096" watchObservedRunningTime="2026-09-10 21:07:11.62537003 +0000 UTC m=+617.443944151" Sep 10 21:07:14.619042 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:14.619001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" event={"ID":"9c873f35-55af-4bd2-af81-f34a6d48fc23","Type":"ContainerStarted","Data":"de1037f70ee7143464deca997bc33906f253141c0c07c513a39ac12d920f3ed3"} Sep 10 21:07:14.619562 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:14.619255 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:14.620559 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:14.620540 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" event={"ID":"ed07e49a-5e29-4489-93fd-6f96bba473db","Type":"ContainerStarted","Data":"5f3a80012b861fba2399758f4730f4e97782db711bdf5ccc81a4e1ce92cb699a"} Sep 10 21:07:14.620669 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:14.620658 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:14.668337 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:14.668272 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" podStartSLOduration=2.010617263 podStartE2EDuration="5.668251658s" totalImagesPullingTime="3.657634395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:14.644627199 +0000 UTC m=+620.463201324" watchObservedRunningTime="2026-09-10 21:07:14.668251658 +0000 UTC m=+620.486825780" Sep 10 21:07:14.668591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:14.668444 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" podStartSLOduration=2.817941702 podStartE2EDuration="5.66843638s" totalImagesPullingTime="2.850494678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:14.66561858 +0000 UTC m=+620.484192702" watchObservedRunningTime="2026-09-10 21:07:14.66843638 +0000 UTC m=+620.487010503" Sep 10 21:07:17.612948 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:17.612861 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:07:18.620721 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.620688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:18.642190 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.642163 2574 generic.go:356] "Generic (PLEG): container finished" podID="58a2ca1c-58c8-445f-a151-3875cc3b4421" containerID="5c70e27c67e7fdc9b224368156ff17b91dfc8474400ea1d488915c823a1ee1a5" exitCode=0 Sep 10 21:07:18.642327 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.642237 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" event={"ID":"58a2ca1c-58c8-445f-a151-3875cc3b4421","Type":"ContainerDied","Data":"5c70e27c67e7fdc9b224368156ff17b91dfc8474400ea1d488915c823a1ee1a5"} Sep 10 21:07:18.642619 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.642605 2574 scope.go:117] "RemoveContainer" containerID="5c70e27c67e7fdc9b224368156ff17b91dfc8474400ea1d488915c823a1ee1a5" Sep 10 21:07:18.643576 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.643536 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:07:18.643648 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.643632 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:07:18.643691 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.643678 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:07:18.739290 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:18.739251 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:19.665000 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:19.664962 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" event={"ID":"58a2ca1c-58c8-445f-a151-3875cc3b4421","Type":"ContainerStarted","Data":"e882f96763258767a609be54a3eab90e26a987e81d8bb6c2575bfcfc58adf9b4"} Sep 10 21:07:19.983741 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:19.983667 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:07:19.984484 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:19.984464 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lf76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:19.984531 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:19.984513 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" podUID="58a2ca1c-58c8-445f-a151-3875cc3b4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Sep 10 21:07:20.984303 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:20.984270 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lf76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:20.984686 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:20.984324 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" podUID="58a2ca1c-58c8-445f-a151-3875cc3b4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Sep 10 21:07:21.983723 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:21.983683 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lf76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:21.983954 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:21.983745 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" podUID="58a2ca1c-58c8-445f-a151-3875cc3b4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Sep 10 21:07:22.984267 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:22.984225 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lf76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:22.984658 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:22.984290 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" podUID="58a2ca1c-58c8-445f-a151-3875cc3b4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Sep 10 21:07:23.984036 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:23.984002 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lf76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Sep 10 21:07:23.984202 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:23.984057 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" podUID="58a2ca1c-58c8-445f-a151-3875cc3b4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Sep 10 21:07:24.091159 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:24.091130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:07:24.603249 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:24.603196 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:24.988012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:24.987933 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:07:24.994902 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:24.994873 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lf76f" Sep 10 21:07:25.402478 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.402449 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77"] Sep 10 21:07:25.405012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.404996 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.407654 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.407621 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-822jm\"" Sep 10 21:07:25.407782 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.407704 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 21:07:25.407782 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.407760 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 21:07:25.407942 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.407844 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 21:07:25.416445 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.416425 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77"] Sep 10 21:07:25.426589 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.426566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:25.508277 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.508245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4l98\" (UniqueName: \"kubernetes.io/projected/7aa718eb-9e98-4588-8f23-957544bce209-kube-api-access-t4l98\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.508446 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.508331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.508446 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.508359 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.508446 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.508373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.508446 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.508389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa718eb-9e98-4588-8f23-957544bce209-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.508581 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.508451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.609756 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.609720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.609945 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.609768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.609945 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.609783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.609945 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.609800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa718eb-9e98-4588-8f23-957544bce209-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.609945 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.609849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.609945 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.609872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4l98\" (UniqueName: \"kubernetes.io/projected/7aa718eb-9e98-4588-8f23-957544bce209-kube-api-access-t4l98\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.610215 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.610188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.610215 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.610203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.610314 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.610275 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.612358 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.612334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa718eb-9e98-4588-8f23-957544bce209-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.612657 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.612638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa718eb-9e98-4588-8f23-957544bce209-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.618347 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.618330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4l98\" (UniqueName: \"kubernetes.io/projected/7aa718eb-9e98-4588-8f23-957544bce209-kube-api-access-t4l98\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77\" (UID: \"7aa718eb-9e98-4588-8f23-957544bce209\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.627971 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.627954 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-nbkbj" Sep 10 21:07:25.628323 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.628296 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-lx7lp" Sep 10 21:07:25.717291 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.717217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:25.996072 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:25.996047 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77"] Sep 10 21:07:25.998129 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:07:25.998095 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa718eb_9e98_4588_8f23_957544bce209.slice/crio-cb235368e328d85735069d9b8330ec0f7cffcb0962ae61013967b54d00956472 WatchSource:0}: Error finding container cb235368e328d85735069d9b8330ec0f7cffcb0962ae61013967b54d00956472: Status 404 returned error can't find the container with id cb235368e328d85735069d9b8330ec0f7cffcb0962ae61013967b54d00956472 Sep 10 21:07:26.704014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:26.703974 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" event={"ID":"7aa718eb-9e98-4588-8f23-957544bce209","Type":"ContainerStarted","Data":"cb235368e328d85735069d9b8330ec0f7cffcb0962ae61013967b54d00956472"} Sep 10 21:07:27.711893 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:27.711857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" event={"ID":"7aa718eb-9e98-4588-8f23-957544bce209","Type":"ContainerStarted","Data":"0ec4e045c37979bf3a0921d6c0d0e6d9f4f78665fedd4891a8d475517ca435ea"} Sep 10 21:07:27.735775 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:27.735727 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" podStartSLOduration=1.752505761 podStartE2EDuration="2.73571083s" totalImagesPullingTime="983.205069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:27.735189413 +0000 UTC m=+633.553763533" watchObservedRunningTime="2026-09-10 21:07:27.73571083 +0000 UTC m=+633.554284951" Sep 10 21:07:29.542831 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:29.539441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:29.617650 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:29.617620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:07:35.718460 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:35.718431 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:35.718907 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:35.718487 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:35.730972 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:35.730949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:35.762148 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:35.762124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77" Sep 10 21:07:42.722493 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:07:42.722452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:08:07.539112 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:07.539064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:08:13.156514 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.156482 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm"] Sep 10 21:08:13.162579 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.162561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.169828 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.169759 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 10 21:08:13.171779 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.171757 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm"] Sep 10 21:08:13.201581 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.201551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:08:13.245635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.245606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.245749 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.245638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.245749 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.245662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zgq\" (UniqueName: \"kubernetes.io/projected/233b67a1-165f-47c5-bf7a-636b0047bc91-kube-api-access-p7zgq\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.245749 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.245716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.245895 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.245797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.245895 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.245856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233b67a1-165f-47c5-bf7a-636b0047bc91-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zgq\" (UniqueName: \"kubernetes.io/projected/233b67a1-165f-47c5-bf7a-636b0047bc91-kube-api-access-p7zgq\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233b67a1-165f-47c5-bf7a-636b0047bc91-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347783 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347783 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347767 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.347899 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.347793 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.349689 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.349668 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/233b67a1-165f-47c5-bf7a-636b0047bc91-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.350056 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.350040 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233b67a1-165f-47c5-bf7a-636b0047bc91-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.355412 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.355389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zgq\" (UniqueName: \"kubernetes.io/projected/233b67a1-165f-47c5-bf7a-636b0047bc91-kube-api-access-p7zgq\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm\" (UID: \"233b67a1-165f-47c5-bf7a-636b0047bc91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.476943 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.476874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:13.751353 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.751330 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm"] Sep 10 21:08:13.753335 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:08:13.753307 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233b67a1_165f_47c5_bf7a_636b0047bc91.slice/crio-24517bdce266eac6131f1a9a2351fb0b5f32dbde8a578642b942f66b77d5130f WatchSource:0}: Error finding container 24517bdce266eac6131f1a9a2351fb0b5f32dbde8a578642b942f66b77d5130f: Status 404 returned error can't find the container with id 24517bdce266eac6131f1a9a2351fb0b5f32dbde8a578642b942f66b77d5130f Sep 10 21:08:13.754693 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.754676 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 21:08:13.939100 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:13.939067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" event={"ID":"233b67a1-165f-47c5-bf7a-636b0047bc91","Type":"ContainerStarted","Data":"24517bdce266eac6131f1a9a2351fb0b5f32dbde8a578642b942f66b77d5130f"} Sep 10 21:08:14.945611 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:14.945576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" event={"ID":"233b67a1-165f-47c5-bf7a-636b0047bc91","Type":"ContainerStarted","Data":"abdfc55b4012a2287e65c4e9b8061cae3cb79e3c8451e133233e56be0127f7e7"} Sep 10 21:08:14.964392 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:14.964344 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" podStartSLOduration=1.7719419589999998 podStartE2EDuration="1.964329436s" totalImagesPullingTime="192.387477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:14.963169886 +0000 UTC m=+680.781744008" watchObservedRunningTime="2026-09-10 21:08:14.964329436 +0000 UTC m=+680.782903556" Sep 10 21:08:23.477258 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:23.477220 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:23.477649 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:23.477309 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:23.489700 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:23.489678 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:23.922314 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:23.922281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:08:24.002486 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:24.002459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm" Sep 10 21:08:37.544584 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:37.544545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:08:57.812938 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.812856 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8"] Sep 10 21:08:57.816591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.816569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:57.819372 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.819351 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 21:08:57.827334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.827310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8"] Sep 10 21:08:57.918472 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.918436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:08:57.956037 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.956004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:57.956037 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.956037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8dp\" (UniqueName: \"kubernetes.io/projected/90bb4c10-1396-4d7d-9066-e713d9d96210-kube-api-access-wv8dp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:57.956284 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.956187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bb4c10-1396-4d7d-9066-e713d9d96210-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:57.956284 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.956228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:57.956396 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.956282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:57.956396 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:57.956319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.056981 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.056949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bb4c10-1396-4d7d-9066-e713d9d96210-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057166 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.056993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057166 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.057033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057166 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.057068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057166 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.057124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057166 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.057148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8dp\" (UniqueName: \"kubernetes.io/projected/90bb4c10-1396-4d7d-9066-e713d9d96210-kube-api-access-wv8dp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057610 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.057580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057721 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.057579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.057721 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.057630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.059603 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.059580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bb4c10-1396-4d7d-9066-e713d9d96210-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.059788 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.059773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bb4c10-1396-4d7d-9066-e713d9d96210-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.067288 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.067226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8dp\" (UniqueName: \"kubernetes.io/projected/90bb4c10-1396-4d7d-9066-e713d9d96210-kube-api-access-wv8dp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-46xp8\" (UID: \"90bb4c10-1396-4d7d-9066-e713d9d96210\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.127981 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.127950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:08:58.402637 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:58.402615 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8"] Sep 10 21:08:58.404201 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:08:58.404173 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bb4c10_1396_4d7d_9066_e713d9d96210.slice/crio-ded02e1e43e827d50968fb96a76b37b6e22e66fe3e3d8885ab4be9bc18ca37cd WatchSource:0}: Error finding container ded02e1e43e827d50968fb96a76b37b6e22e66fe3e3d8885ab4be9bc18ca37cd: Status 404 returned error can't find the container with id ded02e1e43e827d50968fb96a76b37b6e22e66fe3e3d8885ab4be9bc18ca37cd Sep 10 21:08:59.170887 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:59.170853 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" event={"ID":"90bb4c10-1396-4d7d-9066-e713d9d96210","Type":"ContainerStarted","Data":"29e0ba4623ef80aec1eb93b5e6e718cc7a815e36352dfb3c63fd0360b4385c35"} Sep 10 21:08:59.171271 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:59.170908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" event={"ID":"90bb4c10-1396-4d7d-9066-e713d9d96210","Type":"ContainerStarted","Data":"ded02e1e43e827d50968fb96a76b37b6e22e66fe3e3d8885ab4be9bc18ca37cd"} Sep 10 21:08:59.195580 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:08:59.195532 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" podStartSLOduration=1.991863232 podStartE2EDuration="2.195517977s" totalImagesPullingTime="203.654745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:59.193761595 +0000 UTC m=+725.012335718" watchObservedRunningTime="2026-09-10 21:08:59.195517977 +0000 UTC m=+725.014092137" Sep 10 21:09:05.043956 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:09:05.043917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:09:08.128929 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:09:08.128880 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:09:08.129338 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:09:08.128949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:09:08.141451 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:09:08.141424 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:09:08.227911 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:09:08.227884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-46xp8" Sep 10 21:10:02.328382 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:02.328341 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:02.785300 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:02.785267 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:02.884784 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:02.884743 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:30.903168 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:30.903126 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:30.998271 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:30.998235 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:32.037007 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:32.036967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:32.137499 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:32.137438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:35.681417 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:35.681383 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:48.446346 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:48.446314 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:48.577998 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:48.577954 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:48.814199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:48.814148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:49.114948 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:49.114858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:49.845881 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:49.845840 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:49.941044 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:49.941006 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:50.301152 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:50.301118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:55.450533 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:55.450485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:55.557049 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:55.557018 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:56.742053 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.742021 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c"] Sep 10 21:10:56.745635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.745614 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.749540 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.749520 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-04db70-serving-cert\"" Sep 10 21:10:56.764010 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.763990 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c"] Sep 10 21:10:56.810207 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.810178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:56.863725 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.863693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b3ca20-c604-43f1-aefb-e0922108bbe1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.863913 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.863739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cs4\" (UniqueName: \"kubernetes.io/projected/06b3ca20-c604-43f1-aefb-e0922108bbe1-kube-api-access-j5cs4\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.863913 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.863902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-metrics-certs\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.864009 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.863944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.916985 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.916951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:56.965178 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.965140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.965368 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.965289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b3ca20-c604-43f1-aefb-e0922108bbe1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.965368 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:10:56.965308 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-serving-cert: secret "maas-api-e2e-inf-a-04db70-serving-cert" not found Sep 10 21:10:56.965368 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.965338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cs4\" (UniqueName: \"kubernetes.io/projected/06b3ca20-c604-43f1-aefb-e0922108bbe1-kube-api-access-j5cs4\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.965572 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:10:56.965388 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls podName:06b3ca20-c604-43f1-aefb-e0922108bbe1 nodeName:}" failed. No retries permitted until 2026-09-10 21:10:57.465367232 +0000 UTC m=+843.283941335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls") pod "maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" (UID: "06b3ca20-c604-43f1-aefb-e0922108bbe1") : secret "maas-api-e2e-inf-a-04db70-serving-cert" not found Sep 10 21:10:56.965572 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.965484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-metrics-certs\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.966267 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.966244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b3ca20-c604-43f1-aefb-e0922108bbe1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.968645 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.968621 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-metrics-certs\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:56.980891 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:56.980872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cs4\" (UniqueName: \"kubernetes.io/projected/06b3ca20-c604-43f1-aefb-e0922108bbe1-kube-api-access-j5cs4\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:57.073371 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.073341 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:57.423748 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.423665 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:57.471098 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.471063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:57.473595 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.473575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls\") pod \"maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:57.525148 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.525110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:57.658210 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.658176 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:57.856536 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.856505 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:57.947040 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:57.946203 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c"] Sep 10 21:10:57.949763 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:10:57.949738 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b3ca20_c604_43f1_aefb_e0922108bbe1.slice/crio-0ce94f9487366ea70b0af573952d6dd776bb5a3540999c08f94ab2fe928629d3 WatchSource:0}: Error finding container 0ce94f9487366ea70b0af573952d6dd776bb5a3540999c08f94ab2fe928629d3: Status 404 returned error can't find the container with id 0ce94f9487366ea70b0af573952d6dd776bb5a3540999c08f94ab2fe928629d3 Sep 10 21:10:58.158199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:58.158159 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:58.443834 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:58.443783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:58.804168 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:58.804127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" event={"ID":"06b3ca20-c604-43f1-aefb-e0922108bbe1","Type":"ContainerStarted","Data":"af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827"} Sep 10 21:10:58.804371 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:58.804201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" event={"ID":"06b3ca20-c604-43f1-aefb-e0922108bbe1","Type":"ContainerStarted","Data":"0ce94f9487366ea70b0af573952d6dd776bb5a3540999c08f94ab2fe928629d3"} Sep 10 21:10:58.804371 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:58.804241 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:10:59.585337 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:59.585283 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:59.712527 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:59.712493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:10:59.900237 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:10:59.900157 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:00.157387 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.157280 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" podStartSLOduration=3.702709912 podStartE2EDuration="4.15725714s" totalImagesPullingTime="454.547228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:58.856127288 +0000 UTC m=+844.674701411" watchObservedRunningTime="2026-09-10 21:11:00.15725714 +0000 UTC m=+845.975831342" Sep 10 21:11:00.158718 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.158700 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt"] Sep 10 21:11:00.162441 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.162423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.169683 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.169661 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4fafd099-openshift-default-dockercfg-25lpm\"" Sep 10 21:11:00.182446 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.182417 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt"] Sep 10 21:11:00.231340 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.231304 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:00.301650 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-token\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.301882 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-certs\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.301882 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-credential-socket\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.301882 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-envoy\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.301882 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-podinfo\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.301882 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5debe260-2c83-43f4-bc82-b0e78c527b6e-istiod-ca-cert\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.302172 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301921 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-socket\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.302172 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.301950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-data\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.302172 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.302029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k867\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-kube-api-access-2k867\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403063 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-envoy\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-podinfo\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5debe260-2c83-43f4-bc82-b0e78c527b6e-istiod-ca-cert\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-socket\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-data\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403454 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k867\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-kube-api-access-2k867\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403454 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-token\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403454 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-certs\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403644 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403616 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-socket\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.403888 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-credential-socket\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.404027 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5debe260-2c83-43f4-bc82-b0e78c527b6e-istiod-ca-cert\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.404135 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.403998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-data\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.404230 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.404210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-certs\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.404292 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.404247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-credential-socket\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.405497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.405481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-envoy\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.405801 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.405785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-podinfo\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.412319 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.412254 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k867\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-kube-api-access-2k867\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.412319 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.412280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-token\") pod \"e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.472357 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.472325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:00.696982 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.696878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:00.758233 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.758210 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt"] Sep 10 21:11:00.760454 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:11:00.760431 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5debe260_2c83_43f4_bc82_b0e78c527b6e.slice/crio-94320b5fd219bf4550e2c3f5f2880ad879bc140fec54fec02e281e05b6b94028 WatchSource:0}: Error finding container 94320b5fd219bf4550e2c3f5f2880ad879bc140fec54fec02e281e05b6b94028: Status 404 returned error can't find the container with id 94320b5fd219bf4550e2c3f5f2880ad879bc140fec54fec02e281e05b6b94028 Sep 10 21:11:00.763391 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.763357 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:11:00.763487 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.763435 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:11:00.763550 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.763483 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:11:00.816416 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:00.816386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" event={"ID":"5debe260-2c83-43f4-bc82-b0e78c527b6e","Type":"ContainerStarted","Data":"94320b5fd219bf4550e2c3f5f2880ad879bc140fec54fec02e281e05b6b94028"} Sep 10 21:11:01.823776 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:01.823723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" event={"ID":"5debe260-2c83-43f4-bc82-b0e78c527b6e","Type":"ContainerStarted","Data":"b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34"} Sep 10 21:11:01.863593 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:01.863537 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podStartSLOduration=1.863519402 podStartE2EDuration="1.863519402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:01.862014717 +0000 UTC m=+847.680588838" watchObservedRunningTime="2026-09-10 21:11:01.863519402 +0000 UTC m=+847.682093522" Sep 10 21:11:02.472742 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:02.472703 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:02.478509 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:02.478480 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:02.829075 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:02.829046 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:02.829969 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:02.829949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:03.367970 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:03.367935 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:03.474477 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:03.474439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:03.930358 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:03.930327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:04.083854 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.083791 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:04.544993 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.544957 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45"] Sep 10 21:11:04.549108 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.549085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.551477 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.551440 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4fafd099-serving-cert\"" Sep 10 21:11:04.573382 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.573352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45"] Sep 10 21:11:04.596480 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.596451 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:04.646836 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.646777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-metrics-certs\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.647016 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.646901 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfd596-7d01-4392-b0fd-3f855ee45a30-openshift-service-ca\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.647016 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.646954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-maas-api-tls\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.647016 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.646991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdq87\" (UniqueName: \"kubernetes.io/projected/b0dfd596-7d01-4392-b0fd-3f855ee45a30-kube-api-access-hdq87\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.740682 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.740646 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:04.747492 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.747467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-metrics-certs\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.747640 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.747542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfd596-7d01-4392-b0fd-3f855ee45a30-openshift-service-ca\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.747640 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.747566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-maas-api-tls\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.747640 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.747602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdq87\" (UniqueName: \"kubernetes.io/projected/b0dfd596-7d01-4392-b0fd-3f855ee45a30-kube-api-access-hdq87\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.748369 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.748301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfd596-7d01-4392-b0fd-3f855ee45a30-openshift-service-ca\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.750184 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.750162 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-maas-api-tls\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.750279 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.750206 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-metrics-certs\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.763199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.763176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdq87\" (UniqueName: \"kubernetes.io/projected/b0dfd596-7d01-4392-b0fd-3f855ee45a30-kube-api-access-hdq87\") pod \"maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:04.814744 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.814687 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:11:04.848948 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.848895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:04.860294 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:04.860261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:05.152944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.152915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45"] Sep 10 21:11:05.154989 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:11:05.154956 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dfd596_7d01_4392_b0fd_3f855ee45a30.slice/crio-ec12272d3dec484a7ae165c5bff3a983448812fff2547232a81318133886184a WatchSource:0}: Error finding container ec12272d3dec484a7ae165c5bff3a983448812fff2547232a81318133886184a: Status 404 returned error can't find the container with id ec12272d3dec484a7ae165c5bff3a983448812fff2547232a81318133886184a Sep 10 21:11:05.427185 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.427104 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:05.539501 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.539475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:05.638359 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.638326 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:05.731923 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.731801 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:05.845405 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.845375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" event={"ID":"b0dfd596-7d01-4392-b0fd-3f855ee45a30","Type":"ContainerStarted","Data":"c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8"} Sep 10 21:11:05.845594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.845425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" event={"ID":"b0dfd596-7d01-4392-b0fd-3f855ee45a30","Type":"ContainerStarted","Data":"ec12272d3dec484a7ae165c5bff3a983448812fff2547232a81318133886184a"} Sep 10 21:11:05.845594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.845476 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:05.870057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:05.870008 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" podStartSLOduration=1.538902861 podStartE2EDuration="1.869993752s" totalImagesPullingTime="331.090891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:05.868666287 +0000 UTC m=+851.687240410" watchObservedRunningTime="2026-09-10 21:11:05.869993752 +0000 UTC m=+851.688567872" Sep 10 21:11:06.222256 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.222224 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8"] Sep 10 21:11:06.225916 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.225895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.228604 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.228581 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c2de43-serving-cert\"" Sep 10 21:11:06.246681 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.246652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8"] Sep 10 21:11:06.308587 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.308563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:06.364745 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.364709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bzq\" (UniqueName: \"kubernetes.io/projected/845ba69b-7808-4069-866c-b12bce94bf63-kube-api-access-k2bzq\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.364928 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.364874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-maas-api-tls\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.364928 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.364907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-metrics-certs\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.364928 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.364923 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845ba69b-7808-4069-866c-b12bce94bf63-openshift-service-ca\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.419707 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.419670 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:06.466426 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.466395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-maas-api-tls\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.466603 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.466456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-metrics-certs\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.466603 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.466498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845ba69b-7808-4069-866c-b12bce94bf63-openshift-service-ca\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.466603 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.466549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bzq\" (UniqueName: \"kubernetes.io/projected/845ba69b-7808-4069-866c-b12bce94bf63-kube-api-access-k2bzq\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.467293 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.467261 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845ba69b-7808-4069-866c-b12bce94bf63-openshift-service-ca\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.469073 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.469052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-metrics-certs\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.469184 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.469101 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-maas-api-tls\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.475268 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.475213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bzq\" (UniqueName: \"kubernetes.io/projected/845ba69b-7808-4069-866c-b12bce94bf63-kube-api-access-k2bzq\") pod \"maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.538544 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.538515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:06.827634 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.827600 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8"] Sep 10 21:11:06.830589 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:11:06.830564 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845ba69b_7808_4069_866c_b12bce94bf63.slice/crio-31c71d9895145da500dfd89d2ce79489ef62c1e7342117da0317f34d2228fc32 WatchSource:0}: Error finding container 31c71d9895145da500dfd89d2ce79489ef62c1e7342117da0317f34d2228fc32: Status 404 returned error can't find the container with id 31c71d9895145da500dfd89d2ce79489ef62c1e7342117da0317f34d2228fc32 Sep 10 21:11:06.852133 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.852102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" event={"ID":"845ba69b-7808-4069-866c-b12bce94bf63","Type":"ContainerStarted","Data":"31c71d9895145da500dfd89d2ce79489ef62c1e7342117da0317f34d2228fc32"} Sep 10 21:11:06.999856 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:06.999784 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:07.236500 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:07.236464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:07.510289 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:07.510197 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:07.757580 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:07.757543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:07.859538 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:07.859502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" event={"ID":"845ba69b-7808-4069-866c-b12bce94bf63","Type":"ContainerStarted","Data":"c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8"} Sep 10 21:11:07.859716 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:07.859676 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:07.889983 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:07.889935 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" podStartSLOduration=1.574004575 podStartE2EDuration="1.889920263s" totalImagesPullingTime="315.915688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:07.886855647 +0000 UTC m=+853.705429781" watchObservedRunningTime="2026-09-10 21:11:07.889920263 +0000 UTC m=+853.708494384" Sep 10 21:11:08.069406 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:08.069369 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:08.985015 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:08.984978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:10.170480 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:10.170437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:11.857758 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:11.857733 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:12.223749 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:12.223664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:12.674133 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:12.674102 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:12.788825 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:12.788785 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:12.920247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:12.920216 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:13.090831 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:13.088053 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:13.872466 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:13.872437 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:11:18.148761 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:18.148728 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:20.159586 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:20.159542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:20.285249 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:20.285206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:20.384973 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:20.384937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:20.776716 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:20.776668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:21.121669 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:21.121640 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:21.281727 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:21.281698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:23.913384 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:23.913354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:24.591615 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.591578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:24.725357 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.725323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:24.960545 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.960467 2574 generic.go:356] "Generic (PLEG): container finished" podID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerID="b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34" exitCode=0 Sep 10 21:11:24.960545 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.960531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" event={"ID":"5debe260-2c83-43f4-bc82-b0e78c527b6e","Type":"ContainerDied","Data":"b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34"} Sep 10 21:11:24.961120 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.961100 2574 scope.go:117] "RemoveContainer" containerID="b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34" Sep 10 21:11:24.962244 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.962213 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:11:24.962334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.962288 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:11:24.962334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:24.962318 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"} Sep 10 21:11:25.970664 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:25.970628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" event={"ID":"5debe260-2c83-43f4-bc82-b0e78c527b6e","Type":"ContainerStarted","Data":"c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e"} Sep 10 21:11:25.973015 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:25.972989 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5"] Sep 10 21:11:25.975836 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:25.975792 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:25.978614 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:25.978586 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7eaf1f-serving-cert\"" Sep 10 21:11:25.991614 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:25.991589 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5"] Sep 10 21:11:26.073449 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.073398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:26.077331 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.077300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tfh\" (UniqueName: \"kubernetes.io/projected/197f38a2-6b4c-413c-a035-780b05749f45-kube-api-access-k5tfh\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.077593 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.077576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-metrics-certs\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.077774 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.077740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.077861 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.077835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197f38a2-6b4c-413c-a035-780b05749f45-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.178628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.178583 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-metrics-certs\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.179449 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.179425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.179798 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.179779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197f38a2-6b4c-413c-a035-780b05749f45-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.180846 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.180652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197f38a2-6b4c-413c-a035-780b05749f45-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.181024 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:11:26.179678 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-serving-cert: secret "maas-api-e2e-inf-b-7eaf1f-serving-cert" not found Sep 10 21:11:26.181571 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:11:26.181556 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls podName:197f38a2-6b4c-413c-a035-780b05749f45 nodeName:}" failed. No retries permitted until 2026-09-10 21:11:26.681168103 +0000 UTC m=+872.499742215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls") pod "maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" (UID: "197f38a2-6b4c-413c-a035-780b05749f45") : secret "maas-api-e2e-inf-b-7eaf1f-serving-cert" not found Sep 10 21:11:26.181865 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.181845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tfh\" (UniqueName: \"kubernetes.io/projected/197f38a2-6b4c-413c-a035-780b05749f45-kube-api-access-k5tfh\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.182842 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.182583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-metrics-certs\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.201349 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.201326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tfh\" (UniqueName: \"kubernetes.io/projected/197f38a2-6b4c-413c-a035-780b05749f45-kube-api-access-k5tfh\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.224247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.224189 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:26.441288 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.441249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:26.473207 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.473172 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:26.473399 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.473285 2574 patch_prober.go:28] interesting pod/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 10 21:11:26.473399 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.473327 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 10 21:11:26.689441 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.689188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.691934 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.691876 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls\") pod \"maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.886941 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.886912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:26.968586 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:26.968496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:27.132266 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.132234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:27.224767 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.224736 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5"] Sep 10 21:11:27.225690 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:11:27.225667 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197f38a2_6b4c_413c_a035_780b05749f45.slice/crio-0974237cb288aa8d32882edbe4f602ce69ca1eed0068237ca61ef3f80d5e4607 WatchSource:0}: Error finding container 0974237cb288aa8d32882edbe4f602ce69ca1eed0068237ca61ef3f80d5e4607: Status 404 returned error can't find the container with id 0974237cb288aa8d32882edbe4f602ce69ca1eed0068237ca61ef3f80d5e4607 Sep 10 21:11:27.472774 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.472681 2574 patch_prober.go:28] interesting pod/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 10 21:11:27.472774 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.472742 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 10 21:11:27.607406 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.607360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:27.983543 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.983513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" event={"ID":"197f38a2-6b4c-413c-a035-780b05749f45","Type":"ContainerStarted","Data":"0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656"} Sep 10 21:11:27.983722 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.983565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" event={"ID":"197f38a2-6b4c-413c-a035-780b05749f45","Type":"ContainerStarted","Data":"0974237cb288aa8d32882edbe4f602ce69ca1eed0068237ca61ef3f80d5e4607"} Sep 10 21:11:27.983722 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:27.983624 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:28.043943 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:28.043177 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" podStartSLOduration=2.689459383 podStartE2EDuration="3.043155489s" totalImagesPullingTime="353.696106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:28.042992503 +0000 UTC m=+873.861566625" watchObservedRunningTime="2026-09-10 21:11:28.043155489 +0000 UTC m=+873.861729611" Sep 10 21:11:28.472800 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:28.472761 2574 patch_prober.go:28] interesting pod/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 10 21:11:28.473184 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:28.472837 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 10 21:11:28.994237 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:28.994204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:29.472799 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:29.472764 2574 patch_prober.go:28] interesting pod/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 10 21:11:29.473009 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:29.472854 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 10 21:11:30.167979 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:30.167950 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:30.473117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:30.473031 2574 patch_prober.go:28] interesting pod/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 10 21:11:30.473117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:30.473087 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 10 21:11:31.369897 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:31.369864 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:31.477358 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:31.477329 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:31.477755 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:31.477592 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:31.478190 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:31.478174 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:32.797585 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:32.797559 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:33.962373 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:33.962343 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:33.997203 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:33.997174 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:11:34.168283 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:34.168136 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:45.200510 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.200479 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45"] Sep 10 21:11:45.201003 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.200705 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" podUID="b0dfd596-7d01-4392-b0fd-3f855ee45a30" containerName="maas-api" containerID="cri-o://c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8" gracePeriod=30 Sep 10 21:11:45.229880 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.229752 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:45.302785 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.302724 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:45.442311 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.442251 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:45.470862 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.470838 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:45.537885 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.537850 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:45.592782 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.592750 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfd596-7d01-4392-b0fd-3f855ee45a30-openshift-service-ca\") pod \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " Sep 10 21:11:45.592954 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.592863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-maas-api-tls\") pod \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " Sep 10 21:11:45.592954 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.592889 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-metrics-certs\") pod \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " Sep 10 21:11:45.593080 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.593026 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdq87\" (UniqueName: \"kubernetes.io/projected/b0dfd596-7d01-4392-b0fd-3f855ee45a30-kube-api-access-hdq87\") pod \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\" (UID: \"b0dfd596-7d01-4392-b0fd-3f855ee45a30\") " Sep 10 21:11:45.593177 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.593145 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0dfd596-7d01-4392-b0fd-3f855ee45a30-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0dfd596-7d01-4392-b0fd-3f855ee45a30" (UID: "b0dfd596-7d01-4392-b0fd-3f855ee45a30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:11:45.593548 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.593511 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0dfd596-7d01-4392-b0fd-3f855ee45a30-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:45.595373 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.595350 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0dfd596-7d01-4392-b0fd-3f855ee45a30" (UID: "b0dfd596-7d01-4392-b0fd-3f855ee45a30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:11:45.595603 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.595579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0dfd596-7d01-4392-b0fd-3f855ee45a30-kube-api-access-hdq87" (OuterVolumeSpecName: "kube-api-access-hdq87") pod "b0dfd596-7d01-4392-b0fd-3f855ee45a30" (UID: "b0dfd596-7d01-4392-b0fd-3f855ee45a30"). InnerVolumeSpecName "kube-api-access-hdq87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:11:45.595653 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.595617 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0dfd596-7d01-4392-b0fd-3f855ee45a30" (UID: "b0dfd596-7d01-4392-b0fd-3f855ee45a30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:11:45.640956 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.640926 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt"] Sep 10 21:11:45.641145 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.641124 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" containerID="cri-o://c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e" gracePeriod=30 Sep 10 21:11:45.653151 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.653120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:45.694289 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.694262 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:45.694289 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.694285 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dfd596-7d01-4392-b0fd-3f855ee45a30-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:45.694451 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:45.694294 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdq87\" (UniqueName: \"kubernetes.io/projected/b0dfd596-7d01-4392-b0fd-3f855ee45a30-kube-api-access-hdq87\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:46.086243 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.086205 2574 generic.go:356] "Generic (PLEG): container finished" podID="b0dfd596-7d01-4392-b0fd-3f855ee45a30" containerID="c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8" exitCode=0 Sep 10 21:11:46.086407 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.086253 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" Sep 10 21:11:46.086407 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.086291 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" event={"ID":"b0dfd596-7d01-4392-b0fd-3f855ee45a30","Type":"ContainerDied","Data":"c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8"} Sep 10 21:11:46.086407 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.086347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45" event={"ID":"b0dfd596-7d01-4392-b0fd-3f855ee45a30","Type":"ContainerDied","Data":"ec12272d3dec484a7ae165c5bff3a983448812fff2547232a81318133886184a"} Sep 10 21:11:46.086407 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.086370 2574 scope.go:117] "RemoveContainer" containerID="c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8" Sep 10 21:11:46.099848 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.099784 2574 scope.go:117] "RemoveContainer" containerID="c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8" Sep 10 21:11:46.100207 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:11:46.100185 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8\": container with ID starting with c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8 not found: ID does not exist" containerID="c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8" Sep 10 21:11:46.100314 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.100217 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8"} err="failed to get container status \"c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8\": rpc error: code = NotFound desc = could not find container \"c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8\": container with ID starting with c43f09c88b701df56067c799138ea116b676bcb01664ba1298f2285735996fa8 not found: ID does not exist" Sep 10 21:11:46.110323 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.110297 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:46.119348 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.119316 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45"] Sep 10 21:11:46.133627 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.133582 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fafd099-6896bf5b56-9pk45"] Sep 10 21:11:46.189170 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.189095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:46.307174 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.307143 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:46.479864 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.479766 2574 patch_prober.go:28] interesting pod/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 21:11:46.479864 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.479851 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 21:11:46.611888 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.611854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:46.652340 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.652313 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0dfd596-7d01-4392-b0fd-3f855ee45a30" path="/var/lib/kubelet/pods/b0dfd596-7d01-4392-b0fd-3f855ee45a30/volumes" Sep 10 21:11:46.780319 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:46.780287 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:47.740558 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:47.740514 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:49.488583 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.488545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:49.673297 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.673266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:49.778581 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.778537 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:49.790889 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.790859 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4"] Sep 10 21:11:49.791607 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.791590 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0dfd596-7d01-4392-b0fd-3f855ee45a30" containerName="maas-api" Sep 10 21:11:49.791701 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.791610 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0dfd596-7d01-4392-b0fd-3f855ee45a30" containerName="maas-api" Sep 10 21:11:49.791757 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.791713 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0dfd596-7d01-4392-b0fd-3f855ee45a30" containerName="maas-api" Sep 10 21:11:49.795391 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.795372 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.798666 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.798557 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d00f8e2d-serving-cert\"" Sep 10 21:11:49.807454 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.807430 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4"] Sep 10 21:11:49.832984 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.832954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0358323-544e-4b47-8107-dab36c771b8e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.833104 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.833043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.833104 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.833094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-metrics-certs\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.833226 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.833161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj46t\" (UniqueName: \"kubernetes.io/projected/d0358323-544e-4b47-8107-dab36c771b8e-kube-api-access-qj46t\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.875310 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.875281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:49.933746 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.933713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0358323-544e-4b47-8107-dab36c771b8e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.933964 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.933845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.933964 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.933907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-metrics-certs\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.933964 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.933946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj46t\" (UniqueName: \"kubernetes.io/projected/d0358323-544e-4b47-8107-dab36c771b8e-kube-api-access-qj46t\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.934133 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:11:49.934014 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-serving-cert: secret "maas-api-e2e-migrate-d00f8e2d-serving-cert" not found Sep 10 21:11:49.934133 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:11:49.934099 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls podName:d0358323-544e-4b47-8107-dab36c771b8e nodeName:}" failed. No retries permitted until 2026-09-10 21:11:50.434077113 +0000 UTC m=+896.252651217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls") pod "maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" (UID: "d0358323-544e-4b47-8107-dab36c771b8e") : secret "maas-api-e2e-migrate-d00f8e2d-serving-cert" not found Sep 10 21:11:49.934583 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.934557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0358323-544e-4b47-8107-dab36c771b8e-openshift-service-ca\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.936732 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.936706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-metrics-certs\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:49.946802 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:49.946779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj46t\" (UniqueName: \"kubernetes.io/projected/d0358323-544e-4b47-8107-dab36c771b8e-kube-api-access-qj46t\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:50.133626 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.133506 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:50.428162 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.428065 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:50.438678 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.438651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:50.441293 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.441266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls\") pod \"maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:50.536269 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.536207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:11:50.709701 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.709626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:50.907592 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.907570 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:50.944186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944156 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-certs\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944352 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944204 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-envoy\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944352 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944262 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-data\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944352 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5debe260-2c83-43f4-bc82-b0e78c527b6e-istiod-ca-cert\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944352 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944347 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-podinfo\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944372 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-socket\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944412 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-token\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944525 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:50.944588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944554 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-credential-socket\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944795 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944614 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:50.944795 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944652 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k867\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-kube-api-access-2k867\") pod \"5debe260-2c83-43f4-bc82-b0e78c527b6e\" (UID: \"5debe260-2c83-43f4-bc82-b0e78c527b6e\") " Sep 10 21:11:50.944795 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944765 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:50.944997 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.944966 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:50.945245 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.945102 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-socket\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:50.945245 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.945125 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-credential-socket\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:50.945245 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.945139 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-workload-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:50.945245 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.945152 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-data\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:50.945657 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.945517 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5debe260-2c83-43f4-bc82-b0e78c527b6e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:11:50.948459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.948242 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-kube-api-access-2k867" (OuterVolumeSpecName: "kube-api-access-2k867") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "kube-api-access-2k867". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:11:50.948459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.948257 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:11:50.949956 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.949930 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 21:11:50.950057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:50.949952 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5debe260-2c83-43f4-bc82-b0e78c527b6e" (UID: "5debe260-2c83-43f4-bc82-b0e78c527b6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 21:11:51.001344 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.001321 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4"] Sep 10 21:11:51.003777 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:11:51.003753 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0358323_544e_4b47_8107_dab36c771b8e.slice/crio-ce4a5bacb08bbba61cebe7f4e9e60fdf6e4588c40fe978cea85b4302cced35dd WatchSource:0}: Error finding container ce4a5bacb08bbba61cebe7f4e9e60fdf6e4588c40fe978cea85b4302cced35dd: Status 404 returned error can't find the container with id ce4a5bacb08bbba61cebe7f4e9e60fdf6e4588c40fe978cea85b4302cced35dd Sep 10 21:11:51.046113 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.046086 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5debe260-2c83-43f4-bc82-b0e78c527b6e-istiod-ca-cert\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:51.046113 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.046110 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-podinfo\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:51.046258 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.046122 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-token\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:51.046258 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.046135 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k867\" (UniqueName: \"kubernetes.io/projected/5debe260-2c83-43f4-bc82-b0e78c527b6e-kube-api-access-2k867\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:51.046258 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.046147 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5debe260-2c83-43f4-bc82-b0e78c527b6e-istio-envoy\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:11:51.113490 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.113452 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" event={"ID":"d0358323-544e-4b47-8107-dab36c771b8e","Type":"ContainerStarted","Data":"ce4a5bacb08bbba61cebe7f4e9e60fdf6e4588c40fe978cea85b4302cced35dd"} Sep 10 21:11:51.116003 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.115961 2574 generic.go:356] "Generic (PLEG): container finished" podID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerID="c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e" exitCode=0 Sep 10 21:11:51.116121 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.116011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" event={"ID":"5debe260-2c83-43f4-bc82-b0e78c527b6e","Type":"ContainerDied","Data":"c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e"} Sep 10 21:11:51.116121 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.116022 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" Sep 10 21:11:51.116121 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.116057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt" event={"ID":"5debe260-2c83-43f4-bc82-b0e78c527b6e","Type":"ContainerDied","Data":"94320b5fd219bf4550e2c3f5f2880ad879bc140fec54fec02e281e05b6b94028"} Sep 10 21:11:51.116121 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.116079 2574 scope.go:117] "RemoveContainer" containerID="c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e" Sep 10 21:11:51.126333 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.126315 2574 scope.go:117] "RemoveContainer" containerID="b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34" Sep 10 21:11:51.135755 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.135737 2574 scope.go:117] "RemoveContainer" containerID="c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e" Sep 10 21:11:51.136087 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:11:51.136065 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e\": container with ID starting with c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e not found: ID does not exist" containerID="c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e" Sep 10 21:11:51.136155 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.136102 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e"} err="failed to get container status \"c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e\": rpc error: code = NotFound desc = could not find container \"c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e\": container with ID starting with c0c2f02b230c9c0b9437695b3a52ddfb53abbb5446125a5e06c98b8f67a3f82e not found: ID does not exist" Sep 10 21:11:51.136155 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.136125 2574 scope.go:117] "RemoveContainer" containerID="b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34" Sep 10 21:11:51.136444 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:11:51.136421 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34\": container with ID starting with b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34 not found: ID does not exist" containerID="b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34" Sep 10 21:11:51.136503 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.136457 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34"} err="failed to get container status \"b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34\": rpc error: code = NotFound desc = could not find container \"b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34\": container with ID starting with b45408170512416276ec5a429725d1a5e015ffafcda352fa26135fd15dba0f34 not found: ID does not exist" Sep 10 21:11:51.149636 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.149602 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt"] Sep 10 21:11:51.151425 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:51.151402 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4fafd099-openshift-default-85fdf44898-4lrtt"] Sep 10 21:11:52.122500 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:52.122465 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" event={"ID":"d0358323-544e-4b47-8107-dab36c771b8e","Type":"ContainerStarted","Data":"88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25"} Sep 10 21:11:52.122938 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:52.122570 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:11:52.143188 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:52.143140 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" podStartSLOduration=2.772270396 podStartE2EDuration="3.143124744s" totalImagesPullingTime="370.854348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:52.142109479 +0000 UTC m=+897.960683605" watchObservedRunningTime="2026-09-10 21:11:52.143124744 +0000 UTC m=+897.961698898" Sep 10 21:11:52.652021 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:52.651988 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" path="/var/lib/kubelet/pods/5debe260-2c83-43f4-bc82-b0e78c527b6e/volumes" Sep 10 21:11:58.137334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:11:58.137305 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:12:17.740292 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:17.740256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:21.205271 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.205227 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4"] Sep 10 21:12:21.205775 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.205505 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" podUID="d0358323-544e-4b47-8107-dab36c771b8e" containerName="maas-api" containerID="cri-o://88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25" gracePeriod=30 Sep 10 21:12:21.239367 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.239330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:21.334644 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.334616 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:21.432796 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.432745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:21.472900 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.472875 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:12:21.502630 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.502592 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:21.558277 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.558248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-metrics-certs\") pod \"d0358323-544e-4b47-8107-dab36c771b8e\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " Sep 10 21:12:21.558418 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.558308 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj46t\" (UniqueName: \"kubernetes.io/projected/d0358323-544e-4b47-8107-dab36c771b8e-kube-api-access-qj46t\") pod \"d0358323-544e-4b47-8107-dab36c771b8e\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " Sep 10 21:12:21.558418 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.558379 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls\") pod \"d0358323-544e-4b47-8107-dab36c771b8e\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " Sep 10 21:12:21.558556 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.558439 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0358323-544e-4b47-8107-dab36c771b8e-openshift-service-ca\") pod \"d0358323-544e-4b47-8107-dab36c771b8e\" (UID: \"d0358323-544e-4b47-8107-dab36c771b8e\") " Sep 10 21:12:21.558842 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.558795 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0358323-544e-4b47-8107-dab36c771b8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0358323-544e-4b47-8107-dab36c771b8e" (UID: "d0358323-544e-4b47-8107-dab36c771b8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:21.561135 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.561074 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d0358323-544e-4b47-8107-dab36c771b8e" (UID: "d0358323-544e-4b47-8107-dab36c771b8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:12:21.561238 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.561157 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0358323-544e-4b47-8107-dab36c771b8e" (UID: "d0358323-544e-4b47-8107-dab36c771b8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:12:21.561238 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.561162 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0358323-544e-4b47-8107-dab36c771b8e-kube-api-access-qj46t" (OuterVolumeSpecName: "kube-api-access-qj46t") pod "d0358323-544e-4b47-8107-dab36c771b8e" (UID: "d0358323-544e-4b47-8107-dab36c771b8e"). InnerVolumeSpecName "kube-api-access-qj46t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:21.621940 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.621904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:21.660036 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.659998 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0358323-544e-4b47-8107-dab36c771b8e-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:21.660036 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.660035 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:21.660036 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.660047 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj46t\" (UniqueName: \"kubernetes.io/projected/d0358323-544e-4b47-8107-dab36c771b8e-kube-api-access-qj46t\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:21.660250 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:21.660055 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0358323-544e-4b47-8107-dab36c771b8e-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:22.063512 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.063464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:22.303483 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.303452 2574 generic.go:356] "Generic (PLEG): container finished" podID="d0358323-544e-4b47-8107-dab36c771b8e" containerID="88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25" exitCode=0 Sep 10 21:12:22.303883 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.303509 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" Sep 10 21:12:22.303883 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.303577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" event={"ID":"d0358323-544e-4b47-8107-dab36c771b8e","Type":"ContainerDied","Data":"88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25"} Sep 10 21:12:22.303883 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.303624 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4" event={"ID":"d0358323-544e-4b47-8107-dab36c771b8e","Type":"ContainerDied","Data":"ce4a5bacb08bbba61cebe7f4e9e60fdf6e4588c40fe978cea85b4302cced35dd"} Sep 10 21:12:22.303883 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.303651 2574 scope.go:117] "RemoveContainer" containerID="88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25" Sep 10 21:12:22.314870 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.314851 2574 scope.go:117] "RemoveContainer" containerID="88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25" Sep 10 21:12:22.315257 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:12:22.315205 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25\": container with ID starting with 88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25 not found: ID does not exist" containerID="88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25" Sep 10 21:12:22.315257 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.315243 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25"} err="failed to get container status \"88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25\": rpc error: code = NotFound desc = could not find container \"88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25\": container with ID starting with 88f63565481d1b110ad1af2a69d04d7280c659503eae397f71c8427b3dab7a25 not found: ID does not exist" Sep 10 21:12:22.330494 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.330436 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4"] Sep 10 21:12:22.338445 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.338423 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d00f8e2d-5895445667-dwwr4"] Sep 10 21:12:22.389978 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.389945 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:22.595626 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.595539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:22.651238 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.651205 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0358323-544e-4b47-8107-dab36c771b8e" path="/var/lib/kubelet/pods/d0358323-544e-4b47-8107-dab36c771b8e/volumes" Sep 10 21:12:22.737740 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:22.737706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:24.748439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:24.748408 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:24.840234 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:24.840144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:25.924135 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.924106 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc"] Sep 10 21:12:25.924825 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.924790 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" Sep 10 21:12:25.924881 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.924826 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" Sep 10 21:12:25.924881 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.924839 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0358323-544e-4b47-8107-dab36c771b8e" containerName="maas-api" Sep 10 21:12:25.924881 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.924848 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0358323-544e-4b47-8107-dab36c771b8e" containerName="maas-api" Sep 10 21:12:25.924977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.924949 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" Sep 10 21:12:25.924977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.924964 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0358323-544e-4b47-8107-dab36c771b8e" containerName="maas-api" Sep 10 21:12:25.929710 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.929687 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:25.936462 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.936439 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-801c3274-serving-cert\"" Sep 10 21:12:25.947298 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.947272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc"] Sep 10 21:12:25.973936 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:25.973875 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:26.002354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.002322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709d0a2b-188b-4a77-b28e-70ee65f74ec8-openshift-service-ca\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.002354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.002356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtxc\" (UniqueName: \"kubernetes.io/projected/709d0a2b-188b-4a77-b28e-70ee65f74ec8-kube-api-access-nhtxc\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.002538 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.002395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-maas-api-tls\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.002538 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.002517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-metrics-certs\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.085044 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.085011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:26.103697 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.103663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709d0a2b-188b-4a77-b28e-70ee65f74ec8-openshift-service-ca\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.103914 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.103712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtxc\" (UniqueName: \"kubernetes.io/projected/709d0a2b-188b-4a77-b28e-70ee65f74ec8-kube-api-access-nhtxc\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.103914 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.103775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-maas-api-tls\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.103914 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.103895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-metrics-certs\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.104437 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.104404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709d0a2b-188b-4a77-b28e-70ee65f74ec8-openshift-service-ca\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.106429 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.106398 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-metrics-certs\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.106569 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.106552 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-maas-api-tls\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.117614 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.117584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtxc\" (UniqueName: \"kubernetes.io/projected/709d0a2b-188b-4a77-b28e-70ee65f74ec8-kube-api-access-nhtxc\") pod \"maas-api-e2e-ait-801c3274-794789864c-b84jc\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.192489 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.192411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:26.242473 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.242446 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:26.546271 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.546235 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc"] Sep 10 21:12:26.547536 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:12:26.547511 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709d0a2b_188b_4a77_b28e_70ee65f74ec8.slice/crio-f1d3f35f80554b6333700aba00cf3d757e99f9742216532c910dd4e023781c2b WatchSource:0}: Error finding container f1d3f35f80554b6333700aba00cf3d757e99f9742216532c910dd4e023781c2b: Status 404 returned error can't find the container with id f1d3f35f80554b6333700aba00cf3d757e99f9742216532c910dd4e023781c2b Sep 10 21:12:26.680882 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.680840 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:26.779513 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:26.779474 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:27.337274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:27.337231 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" event={"ID":"709d0a2b-188b-4a77-b28e-70ee65f74ec8","Type":"ContainerStarted","Data":"a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1"} Sep 10 21:12:27.337705 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:27.337331 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" event={"ID":"709d0a2b-188b-4a77-b28e-70ee65f74ec8","Type":"ContainerStarted","Data":"f1d3f35f80554b6333700aba00cf3d757e99f9742216532c910dd4e023781c2b"} Sep 10 21:12:27.337705 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:27.337363 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:27.358152 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:27.358102 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" podStartSLOduration=2.015533153 podStartE2EDuration="2.358087634s" totalImagesPullingTime="342.554481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:27.356590421 +0000 UTC m=+933.175164543" watchObservedRunningTime="2026-09-10 21:12:27.358087634 +0000 UTC m=+933.176661756" Sep 10 21:12:30.953751 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:30.953687 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:33.348068 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:33.348043 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:47.263440 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:47.263404 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:56.500633 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.500599 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc"] Sep 10 21:12:56.501180 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.501150 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" podUID="709d0a2b-188b-4a77-b28e-70ee65f74ec8" containerName="maas-api" containerID="cri-o://a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1" gracePeriod=30 Sep 10 21:12:56.535874 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.535840 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:56.638759 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.638724 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:56.737671 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.737640 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:56.758230 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.758178 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:56.806427 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.806398 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709d0a2b-188b-4a77-b28e-70ee65f74ec8-openshift-service-ca\") pod \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " Sep 10 21:12:56.806560 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.806434 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-metrics-certs\") pod \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " Sep 10 21:12:56.806560 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.806482 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhtxc\" (UniqueName: \"kubernetes.io/projected/709d0a2b-188b-4a77-b28e-70ee65f74ec8-kube-api-access-nhtxc\") pod \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " Sep 10 21:12:56.806560 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.806496 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-maas-api-tls\") pod \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\" (UID: \"709d0a2b-188b-4a77-b28e-70ee65f74ec8\") " Sep 10 21:12:56.806831 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.806789 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709d0a2b-188b-4a77-b28e-70ee65f74ec8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "709d0a2b-188b-4a77-b28e-70ee65f74ec8" (UID: "709d0a2b-188b-4a77-b28e-70ee65f74ec8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:12:56.808720 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.808694 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "709d0a2b-188b-4a77-b28e-70ee65f74ec8" (UID: "709d0a2b-188b-4a77-b28e-70ee65f74ec8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:12:56.808882 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.808857 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709d0a2b-188b-4a77-b28e-70ee65f74ec8-kube-api-access-nhtxc" (OuterVolumeSpecName: "kube-api-access-nhtxc") pod "709d0a2b-188b-4a77-b28e-70ee65f74ec8" (UID: "709d0a2b-188b-4a77-b28e-70ee65f74ec8"). InnerVolumeSpecName "kube-api-access-nhtxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:12:56.808986 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.808921 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "709d0a2b-188b-4a77-b28e-70ee65f74ec8" (UID: "709d0a2b-188b-4a77-b28e-70ee65f74ec8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:12:56.834060 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.834017 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:56.908315 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.908283 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709d0a2b-188b-4a77-b28e-70ee65f74ec8-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:56.908315 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.908316 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:56.908548 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.908331 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtxc\" (UniqueName: \"kubernetes.io/projected/709d0a2b-188b-4a77-b28e-70ee65f74ec8-kube-api-access-nhtxc\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:56.908548 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:56.908345 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709d0a2b-188b-4a77-b28e-70ee65f74ec8-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:12:57.046936 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.046897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:57.501231 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.501128 2574 generic.go:356] "Generic (PLEG): container finished" podID="709d0a2b-188b-4a77-b28e-70ee65f74ec8" containerID="a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1" exitCode=0 Sep 10 21:12:57.501231 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.501205 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" Sep 10 21:12:57.501231 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.501196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" event={"ID":"709d0a2b-188b-4a77-b28e-70ee65f74ec8","Type":"ContainerDied","Data":"a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1"} Sep 10 21:12:57.501970 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.501305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc" event={"ID":"709d0a2b-188b-4a77-b28e-70ee65f74ec8","Type":"ContainerDied","Data":"f1d3f35f80554b6333700aba00cf3d757e99f9742216532c910dd4e023781c2b"} Sep 10 21:12:57.501970 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.501341 2574 scope.go:117] "RemoveContainer" containerID="a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1" Sep 10 21:12:57.512608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.512590 2574 scope.go:117] "RemoveContainer" containerID="a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1" Sep 10 21:12:57.512939 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:12:57.512920 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1\": container with ID starting with a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1 not found: ID does not exist" containerID="a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1" Sep 10 21:12:57.513008 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.512947 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1"} err="failed to get container status \"a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1\": rpc error: code = NotFound desc = could not find container \"a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1\": container with ID starting with a117d1cc5ff2a2166f1aa61d6ac949868f7fdafb05db21ba551acc0f7ad24ae1 not found: ID does not exist" Sep 10 21:12:57.527551 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.527524 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc"] Sep 10 21:12:57.531248 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.531227 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-801c3274-794789864c-b84jc"] Sep 10 21:12:57.933830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:57.933781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:58.356956 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:58.356931 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:58.648266 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:58.648170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:12:58.653286 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:58.653252 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709d0a2b-188b-4a77-b28e-70ee65f74ec8" path="/var/lib/kubelet/pods/709d0a2b-188b-4a77-b28e-70ee65f74ec8/volumes" Sep 10 21:12:59.079117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:12:59.079084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:01.063445 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:01.063415 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:01.161332 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:01.161293 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:02.011153 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.011118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:02.111455 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.111426 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:02.142706 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.142672 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j"] Sep 10 21:13:02.143182 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.143168 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" Sep 10 21:13:02.143233 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.143185 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" Sep 10 21:13:02.143233 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.143201 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709d0a2b-188b-4a77-b28e-70ee65f74ec8" containerName="maas-api" Sep 10 21:13:02.143233 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.143208 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="709d0a2b-188b-4a77-b28e-70ee65f74ec8" containerName="maas-api" Sep 10 21:13:02.143325 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.143289 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5debe260-2c83-43f4-bc82-b0e78c527b6e" containerName="istio-proxy" Sep 10 21:13:02.143325 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.143298 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="709d0a2b-188b-4a77-b28e-70ee65f74ec8" containerName="maas-api" Sep 10 21:13:02.147438 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.147409 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.149639 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.149622 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cbc740e6-serving-cert\"" Sep 10 21:13:02.159863 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.159841 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j"] Sep 10 21:13:02.211328 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.211296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:02.262469 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.262372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47dde76-2d0a-4831-b066-c27f5dec58eb-openshift-service-ca\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.262469 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.262434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29rm\" (UniqueName: \"kubernetes.io/projected/c47dde76-2d0a-4831-b066-c27f5dec58eb-kube-api-access-t29rm\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.262698 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.262510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.262698 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.262598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-metrics-certs\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.311587 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.311550 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:02.363864 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.363799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47dde76-2d0a-4831-b066-c27f5dec58eb-openshift-service-ca\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.364062 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.363890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29rm\" (UniqueName: \"kubernetes.io/projected/c47dde76-2d0a-4831-b066-c27f5dec58eb-kube-api-access-t29rm\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.364062 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.363910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.364062 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.363949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-metrics-certs\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.364234 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:13:02.364074 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-serving-cert: secret "maas-api-e2e-derive-cbc740e6-serving-cert" not found Sep 10 21:13:02.364234 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:13:02.364167 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls podName:c47dde76-2d0a-4831-b066-c27f5dec58eb nodeName:}" failed. No retries permitted until 2026-09-10 21:13:02.864145875 +0000 UTC m=+968.682719975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls") pod "maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" (UID: "c47dde76-2d0a-4831-b066-c27f5dec58eb") : secret "maas-api-e2e-derive-cbc740e6-serving-cert" not found Sep 10 21:13:02.364576 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.364557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47dde76-2d0a-4831-b066-c27f5dec58eb-openshift-service-ca\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.366565 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.366545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-metrics-certs\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.372241 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.372218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29rm\" (UniqueName: \"kubernetes.io/projected/c47dde76-2d0a-4831-b066-c27f5dec58eb-kube-api-access-t29rm\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.513401 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.513296 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:02.799354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.799320 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:02.868657 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.868625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.871350 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.871330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls\") pod \"maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:02.907862 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:02.907828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:03.062305 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:03.062233 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:03.338227 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:03.338196 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j"] Sep 10 21:13:03.340037 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:13:03.340012 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47dde76_2d0a_4831_b066_c27f5dec58eb.slice/crio-0722297fd13a5fd76285722d01a748da1e4736d019ed00d663dbddffdebda347 WatchSource:0}: Error finding container 0722297fd13a5fd76285722d01a748da1e4736d019ed00d663dbddffdebda347: Status 404 returned error can't find the container with id 0722297fd13a5fd76285722d01a748da1e4736d019ed00d663dbddffdebda347 Sep 10 21:13:03.535781 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:03.535750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" event={"ID":"c47dde76-2d0a-4831-b066-c27f5dec58eb","Type":"ContainerStarted","Data":"0722297fd13a5fd76285722d01a748da1e4736d019ed00d663dbddffdebda347"} Sep 10 21:13:04.542174 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:04.542132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" event={"ID":"c47dde76-2d0a-4831-b066-c27f5dec58eb","Type":"ContainerStarted","Data":"345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a"} Sep 10 21:13:04.542628 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:04.542238 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:04.563505 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:04.563456 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" podStartSLOduration=2.081335201 podStartE2EDuration="2.563441577s" totalImagesPullingTime="482.106376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:04.560552263 +0000 UTC m=+970.379126383" watchObservedRunningTime="2026-09-10 21:13:04.563441577 +0000 UTC m=+970.382015698" Sep 10 21:13:07.069509 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.069468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:07.704533 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.704502 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8"] Sep 10 21:13:07.704757 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.704726 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" podUID="845ba69b-7808-4069-866c-b12bce94bf63" containerName="maas-api" containerID="cri-o://c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8" gracePeriod=30 Sep 10 21:13:07.740200 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.740163 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:07.816827 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.816689 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:07.897849 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.897794 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:07.962904 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.962845 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:13:07.972597 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:07.972566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:08.020798 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.020772 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-metrics-certs\") pod \"845ba69b-7808-4069-866c-b12bce94bf63\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " Sep 10 21:13:08.020983 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.020897 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-maas-api-tls\") pod \"845ba69b-7808-4069-866c-b12bce94bf63\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " Sep 10 21:13:08.020983 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.020916 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2bzq\" (UniqueName: \"kubernetes.io/projected/845ba69b-7808-4069-866c-b12bce94bf63-kube-api-access-k2bzq\") pod \"845ba69b-7808-4069-866c-b12bce94bf63\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " Sep 10 21:13:08.020983 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.020948 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845ba69b-7808-4069-866c-b12bce94bf63-openshift-service-ca\") pod \"845ba69b-7808-4069-866c-b12bce94bf63\" (UID: \"845ba69b-7808-4069-866c-b12bce94bf63\") " Sep 10 21:13:08.021404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.021371 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845ba69b-7808-4069-866c-b12bce94bf63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "845ba69b-7808-4069-866c-b12bce94bf63" (UID: "845ba69b-7808-4069-866c-b12bce94bf63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:13:08.023199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.023171 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "845ba69b-7808-4069-866c-b12bce94bf63" (UID: "845ba69b-7808-4069-866c-b12bce94bf63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:13:08.023328 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.023304 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845ba69b-7808-4069-866c-b12bce94bf63-kube-api-access-k2bzq" (OuterVolumeSpecName: "kube-api-access-k2bzq") pod "845ba69b-7808-4069-866c-b12bce94bf63" (UID: "845ba69b-7808-4069-866c-b12bce94bf63"). InnerVolumeSpecName "kube-api-access-k2bzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:13:08.023419 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.023399 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "845ba69b-7808-4069-866c-b12bce94bf63" (UID: "845ba69b-7808-4069-866c-b12bce94bf63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:13:08.073833 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.073779 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:08.122251 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.122214 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:08.122251 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.122247 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845ba69b-7808-4069-866c-b12bce94bf63-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:08.122459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.122261 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bzq\" (UniqueName: \"kubernetes.io/projected/845ba69b-7808-4069-866c-b12bce94bf63-kube-api-access-k2bzq\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:08.122459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.122274 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845ba69b-7808-4069-866c-b12bce94bf63-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:08.335883 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.335793 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:08.564056 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.564025 2574 generic.go:356] "Generic (PLEG): container finished" podID="845ba69b-7808-4069-866c-b12bce94bf63" containerID="c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8" exitCode=0 Sep 10 21:13:08.564209 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.564069 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" Sep 10 21:13:08.564209 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.564110 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" event={"ID":"845ba69b-7808-4069-866c-b12bce94bf63","Type":"ContainerDied","Data":"c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8"} Sep 10 21:13:08.564209 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.564163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8" event={"ID":"845ba69b-7808-4069-866c-b12bce94bf63","Type":"ContainerDied","Data":"31c71d9895145da500dfd89d2ce79489ef62c1e7342117da0317f34d2228fc32"} Sep 10 21:13:08.564209 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.564191 2574 scope.go:117] "RemoveContainer" containerID="c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8" Sep 10 21:13:08.573606 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.573591 2574 scope.go:117] "RemoveContainer" containerID="c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8" Sep 10 21:13:08.573870 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:13:08.573849 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8\": container with ID starting with c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8 not found: ID does not exist" containerID="c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8" Sep 10 21:13:08.573934 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.573877 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8"} err="failed to get container status \"c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8\": rpc error: code = NotFound desc = could not find container \"c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8\": container with ID starting with c637d4660b5a1259974c1ffc9d5c6e39032ce2366af583fbffd44fdcd43cd0e8 not found: ID does not exist" Sep 10 21:13:08.593586 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.593527 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8"] Sep 10 21:13:08.595502 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.595483 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c2de43-7bdbc8f885-94rp8"] Sep 10 21:13:08.658075 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:08.658041 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845ba69b-7808-4069-866c-b12bce94bf63" path="/var/lib/kubelet/pods/845ba69b-7808-4069-866c-b12bce94bf63/volumes" Sep 10 21:13:10.553048 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:10.553022 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:13.358229 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:13.358193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:23.206310 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:23.206256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:26.238111 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:26.238073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:28.505566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:28.505524 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:29.362665 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:29.362632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:29.932017 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:29.931986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:33.024521 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.024486 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j"] Sep 10 21:13:33.024962 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.024755 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" podUID="c47dde76-2d0a-4831-b066-c27f5dec58eb" containerName="maas-api" containerID="cri-o://345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a" gracePeriod=30 Sep 10 21:13:33.056691 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.056659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:33.153163 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.146875 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:33.231494 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.231452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:33.300624 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.300602 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:33.331669 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.331600 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:33.410067 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.410033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:33.470447 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.470416 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-metrics-certs\") pod \"c47dde76-2d0a-4831-b066-c27f5dec58eb\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " Sep 10 21:13:33.470610 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.470478 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t29rm\" (UniqueName: \"kubernetes.io/projected/c47dde76-2d0a-4831-b066-c27f5dec58eb-kube-api-access-t29rm\") pod \"c47dde76-2d0a-4831-b066-c27f5dec58eb\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " Sep 10 21:13:33.470610 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.470532 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls\") pod \"c47dde76-2d0a-4831-b066-c27f5dec58eb\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " Sep 10 21:13:33.470610 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.470603 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47dde76-2d0a-4831-b066-c27f5dec58eb-openshift-service-ca\") pod \"c47dde76-2d0a-4831-b066-c27f5dec58eb\" (UID: \"c47dde76-2d0a-4831-b066-c27f5dec58eb\") " Sep 10 21:13:33.470956 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.470927 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47dde76-2d0a-4831-b066-c27f5dec58eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c47dde76-2d0a-4831-b066-c27f5dec58eb" (UID: "c47dde76-2d0a-4831-b066-c27f5dec58eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:13:33.472964 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.472937 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c47dde76-2d0a-4831-b066-c27f5dec58eb" (UID: "c47dde76-2d0a-4831-b066-c27f5dec58eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:13:33.473155 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.473136 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47dde76-2d0a-4831-b066-c27f5dec58eb-kube-api-access-t29rm" (OuterVolumeSpecName: "kube-api-access-t29rm") pod "c47dde76-2d0a-4831-b066-c27f5dec58eb" (UID: "c47dde76-2d0a-4831-b066-c27f5dec58eb"). InnerVolumeSpecName "kube-api-access-t29rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:13:33.473250 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.473232 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c47dde76-2d0a-4831-b066-c27f5dec58eb" (UID: "c47dde76-2d0a-4831-b066-c27f5dec58eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:13:33.507404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.507378 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:33.572055 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.571965 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c47dde76-2d0a-4831-b066-c27f5dec58eb-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:33.572055 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.571994 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:33.572055 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.572003 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29rm\" (UniqueName: \"kubernetes.io/projected/c47dde76-2d0a-4831-b066-c27f5dec58eb-kube-api-access-t29rm\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:33.572055 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.572012 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c47dde76-2d0a-4831-b066-c27f5dec58eb-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:13:33.697332 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.697295 2574 generic.go:356] "Generic (PLEG): container finished" podID="c47dde76-2d0a-4831-b066-c27f5dec58eb" containerID="345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a" exitCode=0 Sep 10 21:13:33.697487 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.697349 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" Sep 10 21:13:33.697487 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.697381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" event={"ID":"c47dde76-2d0a-4831-b066-c27f5dec58eb","Type":"ContainerDied","Data":"345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a"} Sep 10 21:13:33.697487 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.697438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j" event={"ID":"c47dde76-2d0a-4831-b066-c27f5dec58eb","Type":"ContainerDied","Data":"0722297fd13a5fd76285722d01a748da1e4736d019ed00d663dbddffdebda347"} Sep 10 21:13:33.697487 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.697476 2574 scope.go:117] "RemoveContainer" containerID="345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a" Sep 10 21:13:33.707601 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.707586 2574 scope.go:117] "RemoveContainer" containerID="345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a" Sep 10 21:13:33.707930 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:13:33.707906 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a\": container with ID starting with 345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a not found: ID does not exist" containerID="345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a" Sep 10 21:13:33.708050 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.707938 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a"} err="failed to get container status \"345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a\": rpc error: code = NotFound desc = could not find container \"345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a\": container with ID starting with 345b4acb5cfe361bbeba676cf99a9c4dc8ada8b6d50d39d9d92c93e3406d319a not found: ID does not exist" Sep 10 21:13:33.719210 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.719179 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j"] Sep 10 21:13:33.726117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:33.726096 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cbc740e6-5d5574bd96-chc4j"] Sep 10 21:13:34.175232 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:34.175141 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:34.275395 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:34.275361 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:34.652008 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:34.651976 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47dde76-2d0a-4831-b066-c27f5dec58eb" path="/var/lib/kubelet/pods/c47dde76-2d0a-4831-b066-c27f5dec58eb/volumes" Sep 10 21:13:35.659735 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:35.659700 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:40.134697 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:40.134664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:42.719596 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:42.719522 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:42.826579 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:42.826539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:42.921563 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:42.921531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:43.020421 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:43.020327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:43.389869 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:43.389835 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:51.346378 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:51.346347 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:51.446976 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:51.446935 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:51.593857 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:51.593828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:52.260413 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:52.260375 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:52.372900 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:52.372857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:52.764698 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:52.764664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:55.920032 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:55.919989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:56.015723 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:56.015687 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:57.240924 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.240891 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg"] Sep 10 21:13:57.241566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.241425 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47dde76-2d0a-4831-b066-c27f5dec58eb" containerName="maas-api" Sep 10 21:13:57.241566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.241439 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47dde76-2d0a-4831-b066-c27f5dec58eb" containerName="maas-api" Sep 10 21:13:57.241566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.241455 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845ba69b-7808-4069-866c-b12bce94bf63" containerName="maas-api" Sep 10 21:13:57.241566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.241462 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="845ba69b-7808-4069-866c-b12bce94bf63" containerName="maas-api" Sep 10 21:13:57.241566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.241541 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47dde76-2d0a-4831-b066-c27f5dec58eb" containerName="maas-api" Sep 10 21:13:57.241566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.241551 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="845ba69b-7808-4069-866c-b12bce94bf63" containerName="maas-api" Sep 10 21:13:57.256205 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.256172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.259634 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.259608 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg"] Sep 10 21:13:57.262777 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.262755 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-65af3d-serving-cert\"" Sep 10 21:13:57.298135 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.297750 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:57.394956 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.394916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.395269 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.395242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f15fd5f-35ba-413b-914e-abe23e24e9e3-openshift-service-ca\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.395477 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.395450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mhw\" (UniqueName: \"kubernetes.io/projected/9f15fd5f-35ba-413b-914e-abe23e24e9e3-kube-api-access-96mhw\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.395613 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.395525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-metrics-certs\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.399591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.399562 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:57.496394 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.496301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f15fd5f-35ba-413b-914e-abe23e24e9e3-openshift-service-ca\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.496394 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.496358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mhw\" (UniqueName: \"kubernetes.io/projected/9f15fd5f-35ba-413b-914e-abe23e24e9e3-kube-api-access-96mhw\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.496394 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.496391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-metrics-certs\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.496686 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.496439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.496686 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:13:57.496550 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-serving-cert: secret "maas-api-e2e-explicit-65af3d-serving-cert" not found Sep 10 21:13:57.496686 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:13:57.496631 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls podName:9f15fd5f-35ba-413b-914e-abe23e24e9e3 nodeName:}" failed. No retries permitted until 2026-09-10 21:13:57.996608968 +0000 UTC m=+1023.815183094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls") pod "maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" (UID: "9f15fd5f-35ba-413b-914e-abe23e24e9e3") : secret "maas-api-e2e-explicit-65af3d-serving-cert" not found Sep 10 21:13:57.497137 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.497114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f15fd5f-35ba-413b-914e-abe23e24e9e3-openshift-service-ca\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.499226 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.499205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-metrics-certs\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.505621 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.505596 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mhw\" (UniqueName: \"kubernetes.io/projected/9f15fd5f-35ba-413b-914e-abe23e24e9e3-kube-api-access-96mhw\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:57.638688 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.638648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:57.846787 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.846754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:57.951885 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:57.951846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:13:58.002226 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:58.002189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:58.005109 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:58.005082 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls\") pod \"maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:58.183362 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:58.183284 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:13:58.468703 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:58.468674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg"] Sep 10 21:13:58.471209 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:13:58.471183 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f15fd5f_35ba_413b_914e_abe23e24e9e3.slice/crio-3f3509028a8ac552515b17175b796c10b15358690d86aa2075d6371fb291d41e WatchSource:0}: Error finding container 3f3509028a8ac552515b17175b796c10b15358690d86aa2075d6371fb291d41e: Status 404 returned error can't find the container with id 3f3509028a8ac552515b17175b796c10b15358690d86aa2075d6371fb291d41e Sep 10 21:13:58.472452 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:58.472432 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 21:13:58.835394 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:58.835362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" event={"ID":"9f15fd5f-35ba-413b-914e-abe23e24e9e3","Type":"ContainerStarted","Data":"3f3509028a8ac552515b17175b796c10b15358690d86aa2075d6371fb291d41e"} Sep 10 21:13:59.842072 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:59.842033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" event={"ID":"9f15fd5f-35ba-413b-914e-abe23e24e9e3","Type":"ContainerStarted","Data":"779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43"} Sep 10 21:13:59.842452 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:13:59.842198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:14:05.852382 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:05.852355 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:14:05.880071 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:05.880023 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" podStartSLOduration=8.157690195 podStartE2EDuration="8.880008418s" totalImagesPullingTime="722.318223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:59.88482918 +0000 UTC m=+1025.703403291" watchObservedRunningTime="2026-09-10 21:14:05.880008418 +0000 UTC m=+1031.698582539" Sep 10 21:14:09.357921 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:09.357886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:15.414055 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:15.414020 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:15.545577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:15.545531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:15.644911 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:15.644879 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:20.821749 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:20.821705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:24.021175 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:24.021141 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:31.450122 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:31.450078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:36.020371 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:36.020334 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:38.506626 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:38.506585 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:38.657846 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:38.657796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:38.780030 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:38.780001 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:45.899244 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:45.899208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:46.860505 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:46.860471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:47.407310 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:47.407271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:48.214327 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.214296 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg"] Sep 10 21:14:48.214525 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.214486 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" podUID="9f15fd5f-35ba-413b-914e-abe23e24e9e3" containerName="maas-api" containerID="cri-o://779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43" gracePeriod=30 Sep 10 21:14:48.242050 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.242017 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:48.337186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.337154 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:48.440322 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.440291 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:48.474060 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.473996 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:14:48.538973 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.538946 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:48.612287 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.612258 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls\") pod \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " Sep 10 21:14:48.612462 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.612374 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-metrics-certs\") pod \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " Sep 10 21:14:48.612462 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.612422 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f15fd5f-35ba-413b-914e-abe23e24e9e3-openshift-service-ca\") pod \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " Sep 10 21:14:48.612581 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.612516 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96mhw\" (UniqueName: \"kubernetes.io/projected/9f15fd5f-35ba-413b-914e-abe23e24e9e3-kube-api-access-96mhw\") pod \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\" (UID: \"9f15fd5f-35ba-413b-914e-abe23e24e9e3\") " Sep 10 21:14:48.612980 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.612933 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f15fd5f-35ba-413b-914e-abe23e24e9e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f15fd5f-35ba-413b-914e-abe23e24e9e3" (UID: "9f15fd5f-35ba-413b-914e-abe23e24e9e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:14:48.615339 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.615261 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f15fd5f-35ba-413b-914e-abe23e24e9e3" (UID: "9f15fd5f-35ba-413b-914e-abe23e24e9e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:14:48.615339 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.615310 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f15fd5f-35ba-413b-914e-abe23e24e9e3" (UID: "9f15fd5f-35ba-413b-914e-abe23e24e9e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:14:48.615568 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.615437 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f15fd5f-35ba-413b-914e-abe23e24e9e3-kube-api-access-96mhw" (OuterVolumeSpecName: "kube-api-access-96mhw") pod "9f15fd5f-35ba-413b-914e-abe23e24e9e3" (UID: "9f15fd5f-35ba-413b-914e-abe23e24e9e3"). InnerVolumeSpecName "kube-api-access-96mhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:14:48.714274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.714246 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mhw\" (UniqueName: \"kubernetes.io/projected/9f15fd5f-35ba-413b-914e-abe23e24e9e3-kube-api-access-96mhw\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:48.714274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.714272 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:48.714274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.714281 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f15fd5f-35ba-413b-914e-abe23e24e9e3-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:48.714504 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.714289 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f15fd5f-35ba-413b-914e-abe23e24e9e3-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:48.877608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:48.877574 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:49.107757 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.107724 2574 generic.go:356] "Generic (PLEG): container finished" podID="9f15fd5f-35ba-413b-914e-abe23e24e9e3" containerID="779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43" exitCode=0 Sep 10 21:14:49.107922 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.107769 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" Sep 10 21:14:49.107922 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.107825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" event={"ID":"9f15fd5f-35ba-413b-914e-abe23e24e9e3","Type":"ContainerDied","Data":"779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43"} Sep 10 21:14:49.107922 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.107880 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg" event={"ID":"9f15fd5f-35ba-413b-914e-abe23e24e9e3","Type":"ContainerDied","Data":"3f3509028a8ac552515b17175b796c10b15358690d86aa2075d6371fb291d41e"} Sep 10 21:14:49.107922 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.107901 2574 scope.go:117] "RemoveContainer" containerID="779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43" Sep 10 21:14:49.117401 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.117384 2574 scope.go:117] "RemoveContainer" containerID="779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43" Sep 10 21:14:49.117664 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:14:49.117644 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43\": container with ID starting with 779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43 not found: ID does not exist" containerID="779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43" Sep 10 21:14:49.117746 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.117669 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43"} err="failed to get container status \"779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43\": rpc error: code = NotFound desc = could not find container \"779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43\": container with ID starting with 779472b7ff607fbe2ef842e10e68e82357f8ca28177f1a4c24baea23aae07f43 not found: ID does not exist" Sep 10 21:14:49.136069 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.136006 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg"] Sep 10 21:14:49.140058 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.140038 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65af3d-5d47bdf744-zzdkg"] Sep 10 21:14:49.242613 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.242581 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:49.956891 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:49.956860 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:50.653155 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:50.653126 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f15fd5f-35ba-413b-914e-abe23e24e9e3" path="/var/lib/kubelet/pods/9f15fd5f-35ba-413b-914e-abe23e24e9e3/volumes" Sep 10 21:14:55.064471 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:55.064439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:55.188602 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:55.188560 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:55.339668 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:55.339587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:55.770002 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:55.769912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:56.075217 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:56.075179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:56.169452 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:56.169418 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.275217 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.275184 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.403332 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.403302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.503493 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.503448 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.597963 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.597927 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.733452 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.733419 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c"] Sep 10 21:14:57.733692 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.733647 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" podUID="06b3ca20-c604-43f1-aefb-e0922108bbe1" containerName="maas-api" containerID="cri-o://af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827" gracePeriod=30 Sep 10 21:14:57.749632 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.749595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.841850 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.841795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.965571 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.965507 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:57.994341 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:57.994321 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:14:58.101233 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.101191 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-metrics-certs\") pod \"06b3ca20-c604-43f1-aefb-e0922108bbe1\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " Sep 10 21:14:58.101420 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.101247 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls\") pod \"06b3ca20-c604-43f1-aefb-e0922108bbe1\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " Sep 10 21:14:58.101420 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.101361 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5cs4\" (UniqueName: \"kubernetes.io/projected/06b3ca20-c604-43f1-aefb-e0922108bbe1-kube-api-access-j5cs4\") pod \"06b3ca20-c604-43f1-aefb-e0922108bbe1\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " Sep 10 21:14:58.101420 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.101405 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b3ca20-c604-43f1-aefb-e0922108bbe1-openshift-service-ca\") pod \"06b3ca20-c604-43f1-aefb-e0922108bbe1\" (UID: \"06b3ca20-c604-43f1-aefb-e0922108bbe1\") " Sep 10 21:14:58.101725 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.101693 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b3ca20-c604-43f1-aefb-e0922108bbe1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06b3ca20-c604-43f1-aefb-e0922108bbe1" (UID: "06b3ca20-c604-43f1-aefb-e0922108bbe1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:14:58.101871 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.101777 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06b3ca20-c604-43f1-aefb-e0922108bbe1-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:58.103860 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.103772 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b3ca20-c604-43f1-aefb-e0922108bbe1-kube-api-access-j5cs4" (OuterVolumeSpecName: "kube-api-access-j5cs4") pod "06b3ca20-c604-43f1-aefb-e0922108bbe1" (UID: "06b3ca20-c604-43f1-aefb-e0922108bbe1"). InnerVolumeSpecName "kube-api-access-j5cs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:14:58.103860 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.103792 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06b3ca20-c604-43f1-aefb-e0922108bbe1" (UID: "06b3ca20-c604-43f1-aefb-e0922108bbe1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:14:58.104021 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.103929 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "06b3ca20-c604-43f1-aefb-e0922108bbe1" (UID: "06b3ca20-c604-43f1-aefb-e0922108bbe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:14:58.158921 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.158892 2574 generic.go:356] "Generic (PLEG): container finished" podID="06b3ca20-c604-43f1-aefb-e0922108bbe1" containerID="af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827" exitCode=0 Sep 10 21:14:58.159078 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.158944 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" Sep 10 21:14:58.159078 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.158979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" event={"ID":"06b3ca20-c604-43f1-aefb-e0922108bbe1","Type":"ContainerDied","Data":"af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827"} Sep 10 21:14:58.159078 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.159035 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c" event={"ID":"06b3ca20-c604-43f1-aefb-e0922108bbe1","Type":"ContainerDied","Data":"0ce94f9487366ea70b0af573952d6dd776bb5a3540999c08f94ab2fe928629d3"} Sep 10 21:14:58.159078 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.159064 2574 scope.go:117] "RemoveContainer" containerID="af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827" Sep 10 21:14:58.170224 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.170205 2574 scope.go:117] "RemoveContainer" containerID="af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827" Sep 10 21:14:58.170508 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:14:58.170481 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827\": container with ID starting with af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827 not found: ID does not exist" containerID="af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827" Sep 10 21:14:58.170612 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.170509 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827"} err="failed to get container status \"af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827\": rpc error: code = NotFound desc = could not find container \"af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827\": container with ID starting with af1266fad279133ac6acca2c86e15c57fc9793da28d14230464ca7ae58328827 not found: ID does not exist" Sep 10 21:14:58.203060 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.203031 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5cs4\" (UniqueName: \"kubernetes.io/projected/06b3ca20-c604-43f1-aefb-e0922108bbe1-kube-api-access-j5cs4\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:58.203060 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.203057 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:58.203060 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.203066 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06b3ca20-c604-43f1-aefb-e0922108bbe1-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:14:58.204864 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.204786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:58.213912 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.213890 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c"] Sep 10 21:14:58.234175 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.234145 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04db70-6f795469fc-bpw6c"] Sep 10 21:14:58.364222 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.364132 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:58.467111 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.467076 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:58.651051 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.650958 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b3ca20-c604-43f1-aefb-e0922108bbe1" path="/var/lib/kubelet/pods/06b3ca20-c604-43f1-aefb-e0922108bbe1/volumes" Sep 10 21:14:58.670333 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:58.670309 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:14:59.936186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:14:59.936147 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:02.292681 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:02.292634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:02.643936 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:02.643841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:05.512535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:05.512493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:05.589689 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:05.589653 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:05.799823 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:05.799775 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:06.217542 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:06.217445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:06.885007 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:06.884976 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:07.027479 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:07.027443 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:07.548831 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:07.548763 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:09.126156 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:09.126120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:09.231773 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:09.231741 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:09.481578 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:09.481488 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:09.584356 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:09.584323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:13.700496 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:13.700449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:13.801287 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:13.801256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:13.934796 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:13.934761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:14.254544 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:14.254508 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:14.431156 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:14.431095 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:14.650938 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:14.650904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:16.671601 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.671573 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5"] Sep 10 21:15:16.671977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.671744 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" podUID="197f38a2-6b4c-413c-a035-780b05749f45" containerName="maas-api" containerID="cri-o://0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656" gracePeriod=30 Sep 10 21:15:16.684157 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.684124 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:16.766713 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.766677 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:16.836018 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.835986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:16.920847 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.920659 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:16.932355 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.932295 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:15:16.998865 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:16.998829 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:17.086735 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.086699 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls\") pod \"197f38a2-6b4c-413c-a035-780b05749f45\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " Sep 10 21:15:17.086959 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.086742 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197f38a2-6b4c-413c-a035-780b05749f45-openshift-service-ca\") pod \"197f38a2-6b4c-413c-a035-780b05749f45\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " Sep 10 21:15:17.086959 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.086846 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5tfh\" (UniqueName: \"kubernetes.io/projected/197f38a2-6b4c-413c-a035-780b05749f45-kube-api-access-k5tfh\") pod \"197f38a2-6b4c-413c-a035-780b05749f45\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " Sep 10 21:15:17.086959 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.086929 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-metrics-certs\") pod \"197f38a2-6b4c-413c-a035-780b05749f45\" (UID: \"197f38a2-6b4c-413c-a035-780b05749f45\") " Sep 10 21:15:17.087166 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.087139 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197f38a2-6b4c-413c-a035-780b05749f45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "197f38a2-6b4c-413c-a035-780b05749f45" (UID: "197f38a2-6b4c-413c-a035-780b05749f45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:15:17.087302 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.087286 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/197f38a2-6b4c-413c-a035-780b05749f45-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:15:17.089222 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.089196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "197f38a2-6b4c-413c-a035-780b05749f45" (UID: "197f38a2-6b4c-413c-a035-780b05749f45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:15:17.089323 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.089216 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "197f38a2-6b4c-413c-a035-780b05749f45" (UID: "197f38a2-6b4c-413c-a035-780b05749f45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:15:17.089323 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.089302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197f38a2-6b4c-413c-a035-780b05749f45-kube-api-access-k5tfh" (OuterVolumeSpecName: "kube-api-access-k5tfh") pod "197f38a2-6b4c-413c-a035-780b05749f45" (UID: "197f38a2-6b4c-413c-a035-780b05749f45"). InnerVolumeSpecName "kube-api-access-k5tfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:15:17.188511 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.188434 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tfh\" (UniqueName: \"kubernetes.io/projected/197f38a2-6b4c-413c-a035-780b05749f45-kube-api-access-k5tfh\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:15:17.188511 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.188462 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:15:17.188511 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.188472 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/197f38a2-6b4c-413c-a035-780b05749f45-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:15:17.260354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.260307 2574 generic.go:356] "Generic (PLEG): container finished" podID="197f38a2-6b4c-413c-a035-780b05749f45" containerID="0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656" exitCode=0 Sep 10 21:15:17.260354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.260349 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" Sep 10 21:15:17.260566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.260395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" event={"ID":"197f38a2-6b4c-413c-a035-780b05749f45","Type":"ContainerDied","Data":"0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656"} Sep 10 21:15:17.260566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.260446 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5" event={"ID":"197f38a2-6b4c-413c-a035-780b05749f45","Type":"ContainerDied","Data":"0974237cb288aa8d32882edbe4f602ce69ca1eed0068237ca61ef3f80d5e4607"} Sep 10 21:15:17.260566 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.260474 2574 scope.go:117] "RemoveContainer" containerID="0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656" Sep 10 21:15:17.270670 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.270650 2574 scope.go:117] "RemoveContainer" containerID="0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656" Sep 10 21:15:17.271048 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:15:17.271030 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656\": container with ID starting with 0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656 not found: ID does not exist" containerID="0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656" Sep 10 21:15:17.271099 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.271057 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656"} err="failed to get container status \"0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656\": rpc error: code = NotFound desc = could not find container \"0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656\": container with ID starting with 0411b76fda1b9d189f0d99d7965f72b0ec59d4bce1eb566dc52e6cfbff5e9656 not found: ID does not exist" Sep 10 21:15:17.287924 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.287900 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5"] Sep 10 21:15:17.291679 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.291655 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7eaf1f-8fb7485dc-dllf5"] Sep 10 21:15:17.294137 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.294110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:17.891963 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.891931 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:17.992636 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:17.992601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:18.651018 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:18.650987 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="197f38a2-6b4c-413c-a035-780b05749f45" path="/var/lib/kubelet/pods/197f38a2-6b4c-413c-a035-780b05749f45/volumes" Sep 10 21:15:19.117149 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117116 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g"] Sep 10 21:15:19.117560 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117546 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b3ca20-c604-43f1-aefb-e0922108bbe1" containerName="maas-api" Sep 10 21:15:19.117608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117561 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b3ca20-c604-43f1-aefb-e0922108bbe1" containerName="maas-api" Sep 10 21:15:19.117608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117571 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197f38a2-6b4c-413c-a035-780b05749f45" containerName="maas-api" Sep 10 21:15:19.117608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117576 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="197f38a2-6b4c-413c-a035-780b05749f45" containerName="maas-api" Sep 10 21:15:19.117608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117583 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f15fd5f-35ba-413b-914e-abe23e24e9e3" containerName="maas-api" Sep 10 21:15:19.117608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117587 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f15fd5f-35ba-413b-914e-abe23e24e9e3" containerName="maas-api" Sep 10 21:15:19.117760 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117650 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="197f38a2-6b4c-413c-a035-780b05749f45" containerName="maas-api" Sep 10 21:15:19.117760 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117661 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f15fd5f-35ba-413b-914e-abe23e24e9e3" containerName="maas-api" Sep 10 21:15:19.117760 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.117668 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b3ca20-c604-43f1-aefb-e0922108bbe1" containerName="maas-api" Sep 10 21:15:19.122203 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.122181 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.124351 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.124333 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-899bf7e2-serving-cert\"" Sep 10 21:15:19.131186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.131166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g"] Sep 10 21:15:19.197589 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.197549 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:19.207348 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.207321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-metrics-certs\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.207491 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.207398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94f8\" (UniqueName: \"kubernetes.io/projected/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-kube-api-access-g94f8\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.207558 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.207531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-openshift-service-ca\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.207615 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.207587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.300432 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.300369 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:19.308464 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.308437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-openshift-service-ca\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.308588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.308471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.308588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.308500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-metrics-certs\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.308588 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.308553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94f8\" (UniqueName: \"kubernetes.io/projected/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-kube-api-access-g94f8\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.308745 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:15:19.308607 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-serving-cert: secret "maas-api-e2e-mt-899bf7e2-serving-cert" not found Sep 10 21:15:19.308745 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:15:19.308669 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls podName:27ab5a03-3da7-44b9-af7f-04a4c51a39f8 nodeName:}" failed. No retries permitted until 2026-09-10 21:15:19.808649423 +0000 UTC m=+1105.627223529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls") pod "maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" (UID: "27ab5a03-3da7-44b9-af7f-04a4c51a39f8") : secret "maas-api-e2e-mt-899bf7e2-serving-cert" not found Sep 10 21:15:19.309168 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.309138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-openshift-service-ca\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.311332 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.311310 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-metrics-certs\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.317891 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.317871 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94f8\" (UniqueName: \"kubernetes.io/projected/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-kube-api-access-g94f8\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.795180 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.795145 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:19.813356 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.813324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.816150 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.816128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls\") pod \"maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:19.948893 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:19.948704 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:20.032691 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:20.032646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:20.308269 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:20.308234 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g"] Sep 10 21:15:20.311285 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:15:20.311245 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ab5a03_3da7_44b9_af7f_04a4c51a39f8.slice/crio-f927ac30e21be2186d47150e4cdc37aa864a051cb1f010fdecd254b5ae7963c2 WatchSource:0}: Error finding container f927ac30e21be2186d47150e4cdc37aa864a051cb1f010fdecd254b5ae7963c2: Status 404 returned error can't find the container with id f927ac30e21be2186d47150e4cdc37aa864a051cb1f010fdecd254b5ae7963c2 Sep 10 21:15:21.284581 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:21.284544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" event={"ID":"27ab5a03-3da7-44b9-af7f-04a4c51a39f8","Type":"ContainerStarted","Data":"193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671"} Sep 10 21:15:21.284767 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:21.284598 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" event={"ID":"27ab5a03-3da7-44b9-af7f-04a4c51a39f8","Type":"ContainerStarted","Data":"f927ac30e21be2186d47150e4cdc37aa864a051cb1f010fdecd254b5ae7963c2"} Sep 10 21:15:21.284767 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:21.284628 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:24.008441 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:24.008411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:24.081642 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:24.081610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:24.257923 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:24.257890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:24.795764 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:24.795715 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:25.120966 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:25.120878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:25.194549 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:25.194516 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:27.295042 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:27.295011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:15:27.315348 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:27.315295 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" podStartSLOduration=7.930890622 podStartE2EDuration="8.315279578s" totalImagesPullingTime="384.388956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:21.312929154 +0000 UTC m=+1107.131503275" watchObservedRunningTime="2026-09-10 21:15:27.315279578 +0000 UTC m=+1113.133853698" Sep 10 21:15:28.510026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:28.509946 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:28.596842 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:28.596799 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:29.621096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.621068 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v"] Sep 10 21:15:29.624663 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.624644 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.627421 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.627397 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-af7150-serving-cert\"" Sep 10 21:15:29.639528 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.639419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v"] Sep 10 21:15:29.704704 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.704668 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.704870 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.704736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.704870 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.704773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwq8l\" (UniqueName: \"kubernetes.io/projected/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-kube-api-access-wwq8l\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.705006 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.704878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.714483 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.714433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:29.806184 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.806153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.806368 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.806205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.806368 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.806231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwq8l\" (UniqueName: \"kubernetes.io/projected/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-kube-api-access-wwq8l\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.806368 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:15:29.806317 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-serving-cert: secret "maas-api-e2e-shared-a-w3-af7150-serving-cert" not found Sep 10 21:15:29.806368 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.806342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.806572 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:15:29.806395 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls podName:b7e650b6-d3ff-4534-af1e-a9897dc6fd65 nodeName:}" failed. No retries permitted until 2026-09-10 21:15:30.306374085 +0000 UTC m=+1116.124948204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls") pod "maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" (UID: "b7e650b6-d3ff-4534-af1e-a9897dc6fd65") : secret "maas-api-e2e-shared-a-w3-af7150-serving-cert" not found Sep 10 21:15:29.806920 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.806900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.809021 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.808998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.840602 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.840573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwq8l\" (UniqueName: \"kubernetes.io/projected/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-kube-api-access-wwq8l\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:29.853416 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:29.853393 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:30.017058 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:30.016961 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:30.311437 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:30.311400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:30.314089 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:30.314063 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:30.404708 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:30.404676 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:30.506454 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:30.506423 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:30.540725 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:30.540696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:30.827413 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:30.827380 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v"] Sep 10 21:15:30.831256 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:15:30.831230 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e650b6_d3ff_4534_af1e_a9897dc6fd65.slice/crio-eea8179914f95f54ee2cb9a1ff3c29813f90e90e0ed88243e211d48707a9d56a WatchSource:0}: Error finding container eea8179914f95f54ee2cb9a1ff3c29813f90e90e0ed88243e211d48707a9d56a: Status 404 returned error can't find the container with id eea8179914f95f54ee2cb9a1ff3c29813f90e90e0ed88243e211d48707a9d56a Sep 10 21:15:31.335358 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:31.335320 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" event={"ID":"b7e650b6-d3ff-4534-af1e-a9897dc6fd65","Type":"ContainerStarted","Data":"dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb"} Sep 10 21:15:31.335541 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:31.335382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:31.335541 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:31.335403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" event={"ID":"b7e650b6-d3ff-4534-af1e-a9897dc6fd65","Type":"ContainerStarted","Data":"eea8179914f95f54ee2cb9a1ff3c29813f90e90e0ed88243e211d48707a9d56a"} Sep 10 21:15:31.356175 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:31.356126 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" podStartSLOduration=2.052698645 podStartE2EDuration="2.356108702s" totalImagesPullingTime="303.410057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:31.353828615 +0000 UTC m=+1117.172402729" watchObservedRunningTime="2026-09-10 21:15:31.356108702 +0000 UTC m=+1117.174682828" Sep 10 21:15:37.345404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:37.345375 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:15:40.225608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:40.225573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:42.714498 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:42.714462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:43.353471 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:43.353442 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:43.454724 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:43.454685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:43.768916 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:43.768826 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:49.921114 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:49.921077 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:50.021890 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:50.021854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:50.925399 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:50.925364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:51.750977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:51.750937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:52.553069 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:52.553019 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:52.824444 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:52.824358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:53.416929 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:53.416896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:53.734926 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:53.734832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:54.041554 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:54.041511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:54.880551 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:54.880514 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:56.714430 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:56.714392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:56.862464 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:56.862431 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:57.916839 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:57.916787 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7"] Sep 10 21:15:57.922590 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:57.922562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:57.924665 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:57.924644 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7810bd-serving-cert\"" Sep 10 21:15:57.935090 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:57.935063 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7"] Sep 10 21:15:57.955787 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:57.955754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:58.056801 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.056759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:58.073830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.073790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qggkz\" (UniqueName: \"kubernetes.io/projected/4b0a9d2a-45cd-476b-a537-8951622efdeb-kube-api-access-qggkz\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.073994 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.073858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.073994 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.073918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.073994 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.073966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a9d2a-45cd-476b-a537-8951622efdeb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.175026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.174946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.175026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.174977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a9d2a-45cd-476b-a537-8951622efdeb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.175243 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.175033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qggkz\" (UniqueName: \"kubernetes.io/projected/4b0a9d2a-45cd-476b-a537-8951622efdeb-kube-api-access-qggkz\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.175243 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.175062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.175243 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:15:58.175159 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-serving-cert: secret "maas-api-e2e-shared-b-w3-7810bd-serving-cert" not found Sep 10 21:15:58.175243 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:15:58.175224 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls podName:4b0a9d2a-45cd-476b-a537-8951622efdeb nodeName:}" failed. No retries permitted until 2026-09-10 21:15:58.675204634 +0000 UTC m=+1144.493778753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" (UID: "4b0a9d2a-45cd-476b-a537-8951622efdeb") : secret "maas-api-e2e-shared-b-w3-7810bd-serving-cert" not found Sep 10 21:15:58.175881 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.175857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a9d2a-45cd-476b-a537-8951622efdeb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.177640 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.177623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.184303 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.184278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qggkz\" (UniqueName: \"kubernetes.io/projected/4b0a9d2a-45cd-476b-a537-8951622efdeb-kube-api-access-qggkz\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.273422 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.273386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:58.479239 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.479145 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:58.579576 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.579539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:15:58.679096 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.679066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.681722 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.681692 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:58.837857 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:58.837795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:15:59.116159 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:59.116131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7"] Sep 10 21:15:59.117948 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:15:59.117923 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0a9d2a_45cd_476b_a537_8951622efdeb.slice/crio-eb28d95582724a0d7ba43f107758e3cb7fd0402ba0947aa7512987da075a6497 WatchSource:0}: Error finding container eb28d95582724a0d7ba43f107758e3cb7fd0402ba0947aa7512987da075a6497: Status 404 returned error can't find the container with id eb28d95582724a0d7ba43f107758e3cb7fd0402ba0947aa7512987da075a6497 Sep 10 21:15:59.491071 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:15:59.491040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" event={"ID":"4b0a9d2a-45cd-476b-a537-8951622efdeb","Type":"ContainerStarted","Data":"eb28d95582724a0d7ba43f107758e3cb7fd0402ba0947aa7512987da075a6497"} Sep 10 21:16:00.386380 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.386339 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g"] Sep 10 21:16:00.386869 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.386589 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" podUID="27ab5a03-3da7-44b9-af7f-04a4c51a39f8" containerName="maas-api" containerID="cri-o://193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671" gracePeriod=30 Sep 10 21:16:00.415405 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.415377 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:00.489297 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.489237 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:00.510157 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.510118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" event={"ID":"4b0a9d2a-45cd-476b-a537-8951622efdeb","Type":"ContainerStarted","Data":"3d0a2eae847eeb88c8f451b83c1965c2a748eaed19a72ca86a2c7c0f98036458"} Sep 10 21:16:00.510306 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.510259 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:16:00.530734 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.530680 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" podStartSLOduration=3.225335039 podStartE2EDuration="3.53066354s" totalImagesPullingTime="305.328501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:00.528431946 +0000 UTC m=+1146.347006080" watchObservedRunningTime="2026-09-10 21:16:00.53066354 +0000 UTC m=+1146.349237662" Sep 10 21:16:00.561226 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.561138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:00.634205 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.634178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:00.648269 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.648211 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:16:00.701764 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.701698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:00.802891 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.802863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-openshift-service-ca\") pod \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " Sep 10 21:16:00.803089 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.802905 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-metrics-certs\") pod \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " Sep 10 21:16:00.803089 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.802972 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g94f8\" (UniqueName: \"kubernetes.io/projected/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-kube-api-access-g94f8\") pod \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " Sep 10 21:16:00.803089 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.803047 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls\") pod \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\" (UID: \"27ab5a03-3da7-44b9-af7f-04a4c51a39f8\") " Sep 10 21:16:00.803313 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.803284 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27ab5a03-3da7-44b9-af7f-04a4c51a39f8" (UID: "27ab5a03-3da7-44b9-af7f-04a4c51a39f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:16:00.805368 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.805339 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-kube-api-access-g94f8" (OuterVolumeSpecName: "kube-api-access-g94f8") pod "27ab5a03-3da7-44b9-af7f-04a4c51a39f8" (UID: "27ab5a03-3da7-44b9-af7f-04a4c51a39f8"). InnerVolumeSpecName "kube-api-access-g94f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:16:00.805464 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.805366 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "27ab5a03-3da7-44b9-af7f-04a4c51a39f8" (UID: "27ab5a03-3da7-44b9-af7f-04a4c51a39f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:16:00.805464 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.805380 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27ab5a03-3da7-44b9-af7f-04a4c51a39f8" (UID: "27ab5a03-3da7-44b9-af7f-04a4c51a39f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:16:00.903966 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.903872 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94f8\" (UniqueName: \"kubernetes.io/projected/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-kube-api-access-g94f8\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:16:00.903966 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.903912 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:16:00.903966 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.903923 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:16:00.903966 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:00.903932 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ab5a03-3da7-44b9-af7f-04a4c51a39f8-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:16:01.005346 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.005315 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:01.516635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.516603 2574 generic.go:356] "Generic (PLEG): container finished" podID="27ab5a03-3da7-44b9-af7f-04a4c51a39f8" containerID="193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671" exitCode=0 Sep 10 21:16:01.517057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.516645 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" Sep 10 21:16:01.517057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.516680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" event={"ID":"27ab5a03-3da7-44b9-af7f-04a4c51a39f8","Type":"ContainerDied","Data":"193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671"} Sep 10 21:16:01.517057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.516741 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g" event={"ID":"27ab5a03-3da7-44b9-af7f-04a4c51a39f8","Type":"ContainerDied","Data":"f927ac30e21be2186d47150e4cdc37aa864a051cb1f010fdecd254b5ae7963c2"} Sep 10 21:16:01.517057 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.516765 2574 scope.go:117] "RemoveContainer" containerID="193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671" Sep 10 21:16:01.527182 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.527163 2574 scope.go:117] "RemoveContainer" containerID="193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671" Sep 10 21:16:01.527459 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:16:01.527441 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671\": container with ID starting with 193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671 not found: ID does not exist" containerID="193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671" Sep 10 21:16:01.527510 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.527469 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671"} err="failed to get container status \"193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671\": rpc error: code = NotFound desc = could not find container \"193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671\": container with ID starting with 193bac2f7f42b39829cc02e1051adb90f9935050786c724562c39cb76278a671 not found: ID does not exist" Sep 10 21:16:01.538486 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.538463 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g"] Sep 10 21:16:01.542385 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:01.542363 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-899bf7e2-7ffbd5b766-vdq7g"] Sep 10 21:16:02.651144 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:02.651111 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ab5a03-3da7-44b9-af7f-04a4c51a39f8" path="/var/lib/kubelet/pods/27ab5a03-3da7-44b9-af7f-04a4c51a39f8/volumes" Sep 10 21:16:06.522713 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:06.522684 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:16:07.094220 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:07.094186 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:07.195699 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:07.195654 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:07.380097 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:07.380023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:07.719595 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:07.719508 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:07.817686 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:07.817634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:07.941836 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:07.941780 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:14.672255 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:14.672214 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:14.775493 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:14.775456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:14.881779 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:14.881736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:15.078320 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:15.078283 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:21.068744 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:21.068708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:21.135867 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:21.135825 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:21.651749 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:21.651708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:21.962115 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:21.962032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:22.267139 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:22.267046 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:22.711393 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:22.711366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:24.907299 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:24.907247 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:25.006168 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:25.006130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:26.031892 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.031855 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv"] Sep 10 21:16:26.032556 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.032539 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ab5a03-3da7-44b9-af7f-04a4c51a39f8" containerName="maas-api" Sep 10 21:16:26.032600 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.032560 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ab5a03-3da7-44b9-af7f-04a4c51a39f8" containerName="maas-api" Sep 10 21:16:26.032667 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.032656 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ab5a03-3da7-44b9-af7f-04a4c51a39f8" containerName="maas-api" Sep 10 21:16:26.036410 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.036383 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.038856 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.038634 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-9cce25-serving-cert\"" Sep 10 21:16:26.045392 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.045370 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv"] Sep 10 21:16:26.076803 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.076770 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:26.152777 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.152741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.153014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.152841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb96604-4232-4a4d-9347-a5f47e64ab17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.153014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.152905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.153014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.152991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtsj\" (UniqueName: \"kubernetes.io/projected/4eb96604-4232-4a4d-9347-a5f47e64ab17-kube-api-access-rgtsj\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.155906 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.155877 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:26.254130 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.254099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb96604-4232-4a4d-9347-a5f47e64ab17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.254313 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.254155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.254313 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.254206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtsj\" (UniqueName: \"kubernetes.io/projected/4eb96604-4232-4a4d-9347-a5f47e64ab17-kube-api-access-rgtsj\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.254313 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.254247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.254788 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.254763 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb96604-4232-4a4d-9347-a5f47e64ab17-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.256773 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.256755 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.256860 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.256827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.262669 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.262644 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtsj\" (UniqueName: \"kubernetes.io/projected/4eb96604-4232-4a4d-9347-a5f47e64ab17-kube-api-access-rgtsj\") pod \"maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.350573 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.350502 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:26.602963 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.602873 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:26.625957 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.625890 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv"] Sep 10 21:16:26.629157 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:16:26.629129 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb96604_4232_4a4d_9347_a5f47e64ab17.slice/crio-511a0c5f842dac4d5a25511e988216b58d77b02059fa02fb233c2a6030ab8dd9 WatchSource:0}: Error finding container 511a0c5f842dac4d5a25511e988216b58d77b02059fa02fb233c2a6030ab8dd9: Status 404 returned error can't find the container with id 511a0c5f842dac4d5a25511e988216b58d77b02059fa02fb233c2a6030ab8dd9 Sep 10 21:16:26.657129 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.657102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" event={"ID":"4eb96604-4232-4a4d-9347-a5f47e64ab17","Type":"ContainerStarted","Data":"511a0c5f842dac4d5a25511e988216b58d77b02059fa02fb233c2a6030ab8dd9"} Sep 10 21:16:26.719986 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:26.719948 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:27.664034 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:27.664004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" event={"ID":"4eb96604-4232-4a4d-9347-a5f47e64ab17","Type":"ContainerStarted","Data":"95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862"} Sep 10 21:16:27.664458 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:27.664063 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:27.684683 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:27.684639 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" podStartSLOduration=1.296931302 podStartE2EDuration="1.684625998s" totalImagesPullingTime="387.694696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:27.683095829 +0000 UTC m=+1173.501669951" watchObservedRunningTime="2026-09-10 21:16:27.684625998 +0000 UTC m=+1173.503200163" Sep 10 21:16:33.674644 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:33.674613 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:16:35.971256 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:35.971221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:49.695514 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:49.695474 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:49.869232 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:49.869196 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:50.150394 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:50.150352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:50.464465 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:50.464379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:50.858226 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:50.858199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:53.286850 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:53.286772 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:53.392465 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:53.392436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:54.405331 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.405296 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw"] Sep 10 21:16:54.409777 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.409752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.412485 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.412457 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-95d905-serving-cert\"" Sep 10 21:16:54.425052 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.423553 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw"] Sep 10 21:16:54.453779 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.453744 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:54.533958 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.533928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.534157 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.534092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.534157 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.534126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfjd7\" (UniqueName: \"kubernetes.io/projected/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-kube-api-access-rfjd7\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.534286 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.534157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.553262 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.553233 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:54.635203 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.635177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.635345 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.635305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.635345 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.635338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfjd7\" (UniqueName: \"kubernetes.io/projected/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-kube-api-access-rfjd7\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.635461 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.635368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.636693 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.636667 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.638470 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.638435 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-95d905-serving-cert\"" Sep 10 21:16:54.638584 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.638567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.645147 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.645123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfjd7\" (UniqueName: \"kubernetes.io/projected/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-kube-api-access-rfjd7\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:54.645834 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:16:54.645786 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-serving-cert: secret "maas-api-e2e-shared-b-w2-95d905-serving-cert" not found Sep 10 21:16:54.645937 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:16:54.645880 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls podName:a40bc14c-db86-48fb-ae8c-ecd28f7d3541 nodeName:}" failed. No retries permitted until 2026-09-10 21:16:55.145862783 +0000 UTC m=+1200.964436884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls") pod "maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" (UID: "a40bc14c-db86-48fb-ae8c-ecd28f7d3541") : secret "maas-api-e2e-shared-b-w2-95d905-serving-cert" not found Sep 10 21:16:54.736396 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.736302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:54.943666 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:54.943626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:55.041682 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:55.041646 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:16:55.242152 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:55.242115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:55.244757 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:55.244724 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:55.326266 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:55.326190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:55.606309 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:55.606285 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw"] Sep 10 21:16:55.611745 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:16:55.611716 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40bc14c_db86_48fb_ae8c_ecd28f7d3541.slice/crio-7d5a0a9bacd7b4a04f52eaea2c706636f6285933df3b2e4f486fc70283c3c616 WatchSource:0}: Error finding container 7d5a0a9bacd7b4a04f52eaea2c706636f6285933df3b2e4f486fc70283c3c616: Status 404 returned error can't find the container with id 7d5a0a9bacd7b4a04f52eaea2c706636f6285933df3b2e4f486fc70283c3c616 Sep 10 21:16:55.807718 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:55.807680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" event={"ID":"a40bc14c-db86-48fb-ae8c-ecd28f7d3541","Type":"ContainerStarted","Data":"7d5a0a9bacd7b4a04f52eaea2c706636f6285933df3b2e4f486fc70283c3c616"} Sep 10 21:16:56.814974 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:56.814940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" event={"ID":"a40bc14c-db86-48fb-ae8c-ecd28f7d3541","Type":"ContainerStarted","Data":"cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1"} Sep 10 21:16:56.815348 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:56.815004 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:16:56.834934 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:16:56.834893 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" podStartSLOduration=2.500213715 podStartE2EDuration="2.834877506s" totalImagesPullingTime="334.663791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:56.832424392 +0000 UTC m=+1202.650998513" watchObservedRunningTime="2026-09-10 21:16:56.834877506 +0000 UTC m=+1202.653451626" Sep 10 21:17:02.825443 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:02.825411 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:17:06.218991 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:06.218958 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:18.126715 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:18.126664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:26.820866 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:26.820830 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:28.248065 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:28.248026 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:29.552638 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:29.552603 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:29.613729 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:29.613697 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:30.313928 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:30.313897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:30.403633 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:30.403594 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:31.463734 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.463701 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d"] Sep 10 21:17:31.467720 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.467702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.470272 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.470251 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fc0caf1f-serving-cert\"" Sep 10 21:17:31.478430 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.478409 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d"] Sep 10 21:17:31.499442 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.499421 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:31.594775 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.594744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdh47\" (UniqueName: \"kubernetes.io/projected/3de69714-377b-49fc-911b-ae8349f0ac36-kube-api-access-tdh47\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.594987 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.594795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de69714-377b-49fc-911b-ae8349f0ac36-openshift-service-ca\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.594987 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.594843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-metrics-certs\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.594987 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.594953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-maas-api-tls\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.598507 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.598479 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:31.695599 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.695562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de69714-377b-49fc-911b-ae8349f0ac36-openshift-service-ca\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.695599 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.695601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-metrics-certs\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.695873 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.695674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-maas-api-tls\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.695873 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.695769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdh47\" (UniqueName: \"kubernetes.io/projected/3de69714-377b-49fc-911b-ae8349f0ac36-kube-api-access-tdh47\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.696280 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.696251 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de69714-377b-49fc-911b-ae8349f0ac36-openshift-service-ca\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.698495 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.698473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-metrics-certs\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.698602 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.698496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-maas-api-tls\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.699426 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.699403 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:31.704034 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.704014 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdh47\" (UniqueName: \"kubernetes.io/projected/3de69714-377b-49fc-911b-ae8349f0ac36-kube-api-access-tdh47\") pod \"maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.780602 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.780576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:31.998607 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:31.998574 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:32.057536 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:32.057513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d"] Sep 10 21:17:32.059745 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:17:32.059718 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de69714_377b_49fc_911b_ae8349f0ac36.slice/crio-b130fcb8964f093d9bba296c61d84938eb32d9fdfef74124ea48019f477d6bec WatchSource:0}: Error finding container b130fcb8964f093d9bba296c61d84938eb32d9fdfef74124ea48019f477d6bec: Status 404 returned error can't find the container with id b130fcb8964f093d9bba296c61d84938eb32d9fdfef74124ea48019f477d6bec Sep 10 21:17:32.092198 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:32.092168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:32.506052 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:32.505967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:33.017091 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:33.017050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" event={"ID":"3de69714-377b-49fc-911b-ae8349f0ac36","Type":"ContainerStarted","Data":"c1022e6441d72d7057ce1c86d82a62a80a4b54dfb1b3cdbbae1417df425eea66"} Sep 10 21:17:33.017265 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:33.017114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" event={"ID":"3de69714-377b-49fc-911b-ae8349f0ac36","Type":"ContainerStarted","Data":"b130fcb8964f093d9bba296c61d84938eb32d9fdfef74124ea48019f477d6bec"} Sep 10 21:17:33.017265 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:33.017198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:33.038261 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:33.038214 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" podStartSLOduration=1.27447948 podStartE2EDuration="2.038198197s" totalImagesPullingTime="763.718717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:33.035381681 +0000 UTC m=+1238.853955804" watchObservedRunningTime="2026-09-10 21:17:33.038198197 +0000 UTC m=+1238.856772318" Sep 10 21:17:33.341754 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:33.341719 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:39.027242 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:39.027214 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:17:49.586494 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:49.586268 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:17:55.286712 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:17:55.286675 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:02.543105 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:02.543042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:30.156484 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:30.156449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:45.163342 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.163309 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d"] Sep 10 21:18:45.163869 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.163593 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" podUID="3de69714-377b-49fc-911b-ae8349f0ac36" containerName="maas-api" containerID="cri-o://c1022e6441d72d7057ce1c86d82a62a80a4b54dfb1b3cdbbae1417df425eea66" gracePeriod=30 Sep 10 21:18:45.187286 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.187210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:45.286474 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.286437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:45.386091 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.386056 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:45.398182 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.398156 2574 generic.go:356] "Generic (PLEG): container finished" podID="3de69714-377b-49fc-911b-ae8349f0ac36" containerID="c1022e6441d72d7057ce1c86d82a62a80a4b54dfb1b3cdbbae1417df425eea66" exitCode=0 Sep 10 21:18:45.398328 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.398230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" event={"ID":"3de69714-377b-49fc-911b-ae8349f0ac36","Type":"ContainerDied","Data":"c1022e6441d72d7057ce1c86d82a62a80a4b54dfb1b3cdbbae1417df425eea66"} Sep 10 21:18:45.437581 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.437552 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:18:45.497006 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.496960 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:45.515706 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.515674 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-maas-api-tls\") pod \"3de69714-377b-49fc-911b-ae8349f0ac36\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " Sep 10 21:18:45.515878 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.515746 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de69714-377b-49fc-911b-ae8349f0ac36-openshift-service-ca\") pod \"3de69714-377b-49fc-911b-ae8349f0ac36\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " Sep 10 21:18:45.515878 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.515787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdh47\" (UniqueName: \"kubernetes.io/projected/3de69714-377b-49fc-911b-ae8349f0ac36-kube-api-access-tdh47\") pod \"3de69714-377b-49fc-911b-ae8349f0ac36\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " Sep 10 21:18:45.515997 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.515940 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-metrics-certs\") pod \"3de69714-377b-49fc-911b-ae8349f0ac36\" (UID: \"3de69714-377b-49fc-911b-ae8349f0ac36\") " Sep 10 21:18:45.516212 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.516180 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de69714-377b-49fc-911b-ae8349f0ac36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3de69714-377b-49fc-911b-ae8349f0ac36" (UID: "3de69714-377b-49fc-911b-ae8349f0ac36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:18:45.516400 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.516379 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de69714-377b-49fc-911b-ae8349f0ac36-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:18:45.518180 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.518154 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3de69714-377b-49fc-911b-ae8349f0ac36" (UID: "3de69714-377b-49fc-911b-ae8349f0ac36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:18:45.518287 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.518219 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3de69714-377b-49fc-911b-ae8349f0ac36" (UID: "3de69714-377b-49fc-911b-ae8349f0ac36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:18:45.518375 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.518302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de69714-377b-49fc-911b-ae8349f0ac36-kube-api-access-tdh47" (OuterVolumeSpecName: "kube-api-access-tdh47") pod "3de69714-377b-49fc-911b-ae8349f0ac36" (UID: "3de69714-377b-49fc-911b-ae8349f0ac36"). InnerVolumeSpecName "kube-api-access-tdh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:18:45.616916 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.616885 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:18:45.616916 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.616909 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de69714-377b-49fc-911b-ae8349f0ac36-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:18:45.616916 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:45.616918 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdh47\" (UniqueName: \"kubernetes.io/projected/3de69714-377b-49fc-911b-ae8349f0ac36-kube-api-access-tdh47\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:18:46.404903 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:46.404871 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" Sep 10 21:18:46.404903 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:46.404897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d" event={"ID":"3de69714-377b-49fc-911b-ae8349f0ac36","Type":"ContainerDied","Data":"b130fcb8964f093d9bba296c61d84938eb32d9fdfef74124ea48019f477d6bec"} Sep 10 21:18:46.405459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:46.404969 2574 scope.go:117] "RemoveContainer" containerID="c1022e6441d72d7057ce1c86d82a62a80a4b54dfb1b3cdbbae1417df425eea66" Sep 10 21:18:46.432266 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:46.432242 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d"] Sep 10 21:18:46.440481 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:46.440458 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fc0caf1f-5777947684-7bj5d"] Sep 10 21:18:46.650946 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:46.650914 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de69714-377b-49fc-911b-ae8349f0ac36" path="/var/lib/kubelet/pods/3de69714-377b-49fc-911b-ae8349f0ac36/volumes" Sep 10 21:18:48.622349 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:48.622314 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:18:53.643025 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:18:53.642983 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:30.082297 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:30.082261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:30.670389 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:30.670357 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:30.771379 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:30.771333 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:31.090662 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:31.090626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:37.502794 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:37.502754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:37.611767 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:37.611732 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:37.861465 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:37.861420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:39.169488 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:39.169419 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:39.310660 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:39.310629 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:40.095463 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:40.095427 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:40.288253 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:40.288216 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:41.601467 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:41.601424 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:43.084711 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:43.084677 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:43.953267 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:43.953228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:49.331934 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:49.331896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:49.447844 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:49.447792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:50.119527 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:50.119494 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:57.255324 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:57.255288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:57.396924 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:57.396891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:19:57.540409 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:19:57.540374 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:01.545031 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:01.544995 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:08.015478 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:08.015440 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:08.265550 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:08.265465 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:29.187408 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:29.187366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:33.739395 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:33.739360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:37.395854 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:37.395780 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:37.578255 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:37.578216 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:37.769393 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:37.769316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:38.430501 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:38.430466 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:39.257580 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:39.257545 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:41.808476 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:41.808439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:42.023246 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.023212 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:42.912385 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.912351 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc"] Sep 10 21:20:42.913102 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.913080 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de69714-377b-49fc-911b-ae8349f0ac36" containerName="maas-api" Sep 10 21:20:42.913176 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.913107 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de69714-377b-49fc-911b-ae8349f0ac36" containerName="maas-api" Sep 10 21:20:42.913261 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.913248 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de69714-377b-49fc-911b-ae8349f0ac36" containerName="maas-api" Sep 10 21:20:42.917110 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.917084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:42.925577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.925166 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e074ad-serving-cert\"" Sep 10 21:20:42.941051 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:42.941026 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc"] Sep 10 21:20:43.002601 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.002566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:43.062228 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.062191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.062405 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.062309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.062405 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.062375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-metrics-certs\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.062497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.062415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6js2c\" (UniqueName: \"kubernetes.io/projected/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-kube-api-access-6js2c\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.101237 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.101207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:43.163332 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.163258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.163332 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.163314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-metrics-certs\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.163545 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.163349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6js2c\" (UniqueName: \"kubernetes.io/projected/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-kube-api-access-6js2c\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.163545 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.163384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.163545 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:20:43.163435 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-serving-cert: secret "maas-api-e2e-api-a-e074ad-serving-cert" not found Sep 10 21:20:43.163545 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:20:43.163525 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls podName:f9cfbe97-0305-4e55-9628-fff0ec78dd1c nodeName:}" failed. No retries permitted until 2026-09-10 21:20:43.663502911 +0000 UTC m=+1429.482077013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls") pod "maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" (UID: "f9cfbe97-0305-4e55-9628-fff0ec78dd1c") : secret "maas-api-e2e-api-a-e074ad-serving-cert" not found Sep 10 21:20:43.164042 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.164023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.166151 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.166122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-metrics-certs\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.175955 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.175929 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6js2c\" (UniqueName: \"kubernetes.io/projected/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-kube-api-access-6js2c\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.307257 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.307220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:43.553927 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.553885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:43.657028 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.656999 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:20:43.667534 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.667509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.670039 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.670010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls\") pod \"maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:43.829320 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:43.829217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:44.141394 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:44.141369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc"] Sep 10 21:20:44.143733 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:20:44.143707 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cfbe97_0305_4e55_9628_fff0ec78dd1c.slice/crio-c5dd2ad8e9d676a98bf97f0ece464bce328c17f5d6a848c67fd3cb1620366ed1 WatchSource:0}: Error finding container c5dd2ad8e9d676a98bf97f0ece464bce328c17f5d6a848c67fd3cb1620366ed1: Status 404 returned error can't find the container with id c5dd2ad8e9d676a98bf97f0ece464bce328c17f5d6a848c67fd3cb1620366ed1 Sep 10 21:20:44.145074 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:44.145057 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 21:20:45.032724 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:45.032686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" event={"ID":"f9cfbe97-0305-4e55-9628-fff0ec78dd1c","Type":"ContainerStarted","Data":"c4425662a7746960662c17693abaa1f4e1561d3600dd0c0c7190be7c7fbd2f05"} Sep 10 21:20:45.032950 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:45.032746 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" event={"ID":"f9cfbe97-0305-4e55-9628-fff0ec78dd1c","Type":"ContainerStarted","Data":"c5dd2ad8e9d676a98bf97f0ece464bce328c17f5d6a848c67fd3cb1620366ed1"} Sep 10 21:20:45.032950 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:45.032785 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:45.055260 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:45.055208 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" podStartSLOduration=2.650157967 podStartE2EDuration="3.055191335s" totalImagesPullingTime="405.033368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:45.053707586 +0000 UTC m=+1430.872281708" watchObservedRunningTime="2026-09-10 21:20:45.055191335 +0000 UTC m=+1430.873765455" Sep 10 21:20:51.044148 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:51.044058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:20:51.882542 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:20:51.882500 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:05.888383 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:05.888342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:11.660844 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:11.660782 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:12.093287 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:12.093252 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:12.544669 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:12.544637 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:12.769014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:12.768976 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:15.432364 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:15.432318 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:15.978921 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:15.978882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:16.075185 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:16.075142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:16.623223 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:16.623180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:17.223789 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.223760 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r"] Sep 10 21:21:17.227608 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.227593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.235348 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.235325 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d55386-serving-cert\"" Sep 10 21:21:17.284929 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.284893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-metrics-certs\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.285112 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.284954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-maas-api-tls\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.285112 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.285007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84278190-4cf5-45ad-a059-c0539f0506d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.285112 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.285100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpkl2\" (UniqueName: \"kubernetes.io/projected/84278190-4cf5-45ad-a059-c0539f0506d2-kube-api-access-mpkl2\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.317612 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.317581 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r"] Sep 10 21:21:17.386037 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.386004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84278190-4cf5-45ad-a059-c0539f0506d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.386252 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.386077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkl2\" (UniqueName: \"kubernetes.io/projected/84278190-4cf5-45ad-a059-c0539f0506d2-kube-api-access-mpkl2\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.386252 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.386187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-metrics-certs\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.386390 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.386266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-maas-api-tls\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.386878 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.386855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84278190-4cf5-45ad-a059-c0539f0506d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.389063 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.389038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-metrics-certs\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.389219 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.389038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-maas-api-tls\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.394622 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.394579 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:17.406633 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.406604 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkl2\" (UniqueName: \"kubernetes.io/projected/84278190-4cf5-45ad-a059-c0539f0506d2-kube-api-access-mpkl2\") pod \"maas-api-e2e-api-b-d55386-789c9497b7-5xh2r\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.538543 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.538507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:17.598914 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.598722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:17.823844 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:17.823800 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r"] Sep 10 21:21:17.825920 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:21:17.825894 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84278190_4cf5_45ad_a059_c0539f0506d2.slice/crio-d9aa69b224417c3c1959ea7d3aa184e548a8754cfcbf5c2c3fd205a72a20dc64 WatchSource:0}: Error finding container d9aa69b224417c3c1959ea7d3aa184e548a8754cfcbf5c2c3fd205a72a20dc64: Status 404 returned error can't find the container with id d9aa69b224417c3c1959ea7d3aa184e548a8754cfcbf5c2c3fd205a72a20dc64 Sep 10 21:21:18.187471 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:18.187379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:18.212580 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:18.212553 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" event={"ID":"84278190-4cf5-45ad-a059-c0539f0506d2","Type":"ContainerStarted","Data":"d9aa69b224417c3c1959ea7d3aa184e548a8754cfcbf5c2c3fd205a72a20dc64"} Sep 10 21:21:18.285046 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:18.285012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:19.219870 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:19.219833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" event={"ID":"84278190-4cf5-45ad-a059-c0539f0506d2","Type":"ContainerStarted","Data":"3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea"} Sep 10 21:21:19.220255 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:19.219901 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:19.251609 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:19.251562 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" podStartSLOduration=1.904774451 podStartE2EDuration="2.251547434s" totalImagesPullingTime="346.772983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:19.250098703 +0000 UTC m=+1465.068672827" watchObservedRunningTime="2026-09-10 21:21:19.251547434 +0000 UTC m=+1465.070121556" Sep 10 21:21:20.579481 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:20.579445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:25.235721 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:25.235694 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:21:38.851132 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:38.851093 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:40.856047 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:40.856013 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:43.341879 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:43.341845 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:43.831990 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:43.831953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:47.659171 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:47.659136 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:49.025518 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:49.025485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:54.031470 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:54.031429 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:57.556535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:57.556498 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:21:57.657290 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:21:57.657255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:00.212426 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.212390 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc"] Sep 10 21:22:00.212921 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.212652 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" podUID="f9cfbe97-0305-4e55-9628-fff0ec78dd1c" containerName="maas-api" containerID="cri-o://c4425662a7746960662c17693abaa1f4e1561d3600dd0c0c7190be7c7fbd2f05" gracePeriod=30 Sep 10 21:22:00.244901 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.244869 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:00.369574 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.369528 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:00.446991 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.446954 2574 generic.go:356] "Generic (PLEG): container finished" podID="f9cfbe97-0305-4e55-9628-fff0ec78dd1c" containerID="c4425662a7746960662c17693abaa1f4e1561d3600dd0c0c7190be7c7fbd2f05" exitCode=0 Sep 10 21:22:00.447148 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.447002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" event={"ID":"f9cfbe97-0305-4e55-9628-fff0ec78dd1c","Type":"ContainerDied","Data":"c4425662a7746960662c17693abaa1f4e1561d3600dd0c0c7190be7c7fbd2f05"} Sep 10 21:22:00.467635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.467566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:00.472526 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.472506 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:22:00.560686 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.560634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:00.616589 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.616551 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-metrics-certs\") pod \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " Sep 10 21:22:00.616773 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.616619 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls\") pod \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " Sep 10 21:22:00.616773 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.616740 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6js2c\" (UniqueName: \"kubernetes.io/projected/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-kube-api-access-6js2c\") pod \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " Sep 10 21:22:00.616901 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.616833 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-openshift-service-ca\") pod \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\" (UID: \"f9cfbe97-0305-4e55-9628-fff0ec78dd1c\") " Sep 10 21:22:00.617227 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.617182 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9cfbe97-0305-4e55-9628-fff0ec78dd1c" (UID: "f9cfbe97-0305-4e55-9628-fff0ec78dd1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:22:00.619229 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.619202 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9cfbe97-0305-4e55-9628-fff0ec78dd1c" (UID: "f9cfbe97-0305-4e55-9628-fff0ec78dd1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:22:00.619514 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.619489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f9cfbe97-0305-4e55-9628-fff0ec78dd1c" (UID: "f9cfbe97-0305-4e55-9628-fff0ec78dd1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:22:00.619640 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.619504 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-kube-api-access-6js2c" (OuterVolumeSpecName: "kube-api-access-6js2c") pod "f9cfbe97-0305-4e55-9628-fff0ec78dd1c" (UID: "f9cfbe97-0305-4e55-9628-fff0ec78dd1c"). InnerVolumeSpecName "kube-api-access-6js2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:22:00.718517 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.718435 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.718517 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.718464 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.718517 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.718473 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.718517 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.718481 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6js2c\" (UniqueName: \"kubernetes.io/projected/f9cfbe97-0305-4e55-9628-fff0ec78dd1c-kube-api-access-6js2c\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:00.895445 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:00.895406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:01.453855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:01.453801 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" Sep 10 21:22:01.453855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:01.453846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc" event={"ID":"f9cfbe97-0305-4e55-9628-fff0ec78dd1c","Type":"ContainerDied","Data":"c5dd2ad8e9d676a98bf97f0ece464bce328c17f5d6a848c67fd3cb1620366ed1"} Sep 10 21:22:01.454304 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:01.453910 2574 scope.go:117] "RemoveContainer" containerID="c4425662a7746960662c17693abaa1f4e1561d3600dd0c0c7190be7c7fbd2f05" Sep 10 21:22:01.472075 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:01.472052 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc"] Sep 10 21:22:01.475828 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:01.475792 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e074ad-59bf766f64-x7fbc"] Sep 10 21:22:02.650987 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:02.650956 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9cfbe97-0305-4e55-9628-fff0ec78dd1c" path="/var/lib/kubelet/pods/f9cfbe97-0305-4e55-9628-fff0ec78dd1c/volumes" Sep 10 21:22:07.312666 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:07.312630 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:07.412935 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:07.412895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:07.647530 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:07.647447 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:08.052489 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:08.052458 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:13.827872 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:13.827841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:14.241399 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:14.241321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:14.305633 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:14.305597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:15.906562 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:15.906523 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:18.234245 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:18.234206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:23.832312 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:23.832280 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r"] Sep 10 21:22:23.832740 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:23.832563 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" podUID="84278190-4cf5-45ad-a059-c0539f0506d2" containerName="maas-api" containerID="cri-o://3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea" gracePeriod=30 Sep 10 21:22:23.868921 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:23.868887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:23.967636 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:23.967597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:24.066410 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.066378 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:24.084964 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.084908 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:22:24.149129 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.149098 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84278190-4cf5-45ad-a059-c0539f0506d2-openshift-service-ca\") pod \"84278190-4cf5-45ad-a059-c0539f0506d2\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " Sep 10 21:22:24.149308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.149173 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-maas-api-tls\") pod \"84278190-4cf5-45ad-a059-c0539f0506d2\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " Sep 10 21:22:24.149308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.149213 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-metrics-certs\") pod \"84278190-4cf5-45ad-a059-c0539f0506d2\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " Sep 10 21:22:24.149308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.149254 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpkl2\" (UniqueName: \"kubernetes.io/projected/84278190-4cf5-45ad-a059-c0539f0506d2-kube-api-access-mpkl2\") pod \"84278190-4cf5-45ad-a059-c0539f0506d2\" (UID: \"84278190-4cf5-45ad-a059-c0539f0506d2\") " Sep 10 21:22:24.149557 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.149534 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84278190-4cf5-45ad-a059-c0539f0506d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84278190-4cf5-45ad-a059-c0539f0506d2" (UID: "84278190-4cf5-45ad-a059-c0539f0506d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:22:24.151425 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.151401 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84278190-4cf5-45ad-a059-c0539f0506d2" (UID: "84278190-4cf5-45ad-a059-c0539f0506d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:22:24.151547 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.151524 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84278190-4cf5-45ad-a059-c0539f0506d2" (UID: "84278190-4cf5-45ad-a059-c0539f0506d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:22:24.151594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.151579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84278190-4cf5-45ad-a059-c0539f0506d2-kube-api-access-mpkl2" (OuterVolumeSpecName: "kube-api-access-mpkl2") pod "84278190-4cf5-45ad-a059-c0539f0506d2" (UID: "84278190-4cf5-45ad-a059-c0539f0506d2"). InnerVolumeSpecName "kube-api-access-mpkl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:22:24.167000 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.166547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:24.250965 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.250928 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:24.250965 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.250959 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpkl2\" (UniqueName: \"kubernetes.io/projected/84278190-4cf5-45ad-a059-c0539f0506d2-kube-api-access-mpkl2\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:24.250965 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.250971 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84278190-4cf5-45ad-a059-c0539f0506d2-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:24.251201 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.250980 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84278190-4cf5-45ad-a059-c0539f0506d2-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:24.453734 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.453652 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:24.594321 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.594291 2574 generic.go:356] "Generic (PLEG): container finished" podID="84278190-4cf5-45ad-a059-c0539f0506d2" containerID="3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea" exitCode=0 Sep 10 21:22:24.594490 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.594333 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" Sep 10 21:22:24.594490 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.594378 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" event={"ID":"84278190-4cf5-45ad-a059-c0539f0506d2","Type":"ContainerDied","Data":"3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea"} Sep 10 21:22:24.594490 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.594432 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r" event={"ID":"84278190-4cf5-45ad-a059-c0539f0506d2","Type":"ContainerDied","Data":"d9aa69b224417c3c1959ea7d3aa184e548a8754cfcbf5c2c3fd205a72a20dc64"} Sep 10 21:22:24.594490 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.594461 2574 scope.go:117] "RemoveContainer" containerID="3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea" Sep 10 21:22:24.604062 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.604039 2574 scope.go:117] "RemoveContainer" containerID="3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea" Sep 10 21:22:24.604349 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:22:24.604331 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea\": container with ID starting with 3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea not found: ID does not exist" containerID="3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea" Sep 10 21:22:24.604404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.604360 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea"} err="failed to get container status \"3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea\": rpc error: code = NotFound desc = could not find container \"3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea\": container with ID starting with 3310e04e3a5a51273b3506fbbbf57db1b21d9914fa653183b626a3fa4aba25ea not found: ID does not exist" Sep 10 21:22:24.627286 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.627257 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r"] Sep 10 21:22:24.638218 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.638192 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d55386-789c9497b7-5xh2r"] Sep 10 21:22:24.651299 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:24.651274 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84278190-4cf5-45ad-a059-c0539f0506d2" path="/var/lib/kubelet/pods/84278190-4cf5-45ad-a059-c0539f0506d2/volumes" Sep 10 21:22:30.733736 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:30.733699 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:32.415139 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:32.415100 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:33.456311 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:33.456281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:41.477100 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:41.477042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:42.148426 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.148392 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv"] Sep 10 21:22:42.148677 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.148653 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" podUID="4eb96604-4232-4a4d-9347-a5f47e64ab17" containerName="maas-api" containerID="cri-o://95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862" gracePeriod=30 Sep 10 21:22:42.184010 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.183978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:42.286854 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.286802 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:42.385859 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.385821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:42.412519 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.412461 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:22:42.523400 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.523367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:42.529330 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.529306 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgtsj\" (UniqueName: \"kubernetes.io/projected/4eb96604-4232-4a4d-9347-a5f47e64ab17-kube-api-access-rgtsj\") pod \"4eb96604-4232-4a4d-9347-a5f47e64ab17\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " Sep 10 21:22:42.529479 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.529404 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb96604-4232-4a4d-9347-a5f47e64ab17-openshift-service-ca\") pod \"4eb96604-4232-4a4d-9347-a5f47e64ab17\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " Sep 10 21:22:42.529544 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.529507 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-maas-api-tls\") pod \"4eb96604-4232-4a4d-9347-a5f47e64ab17\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " Sep 10 21:22:42.529603 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.529565 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-metrics-certs\") pod \"4eb96604-4232-4a4d-9347-a5f47e64ab17\" (UID: \"4eb96604-4232-4a4d-9347-a5f47e64ab17\") " Sep 10 21:22:42.529830 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.529783 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb96604-4232-4a4d-9347-a5f47e64ab17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4eb96604-4232-4a4d-9347-a5f47e64ab17" (UID: "4eb96604-4232-4a4d-9347-a5f47e64ab17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:22:42.529913 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.529899 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb96604-4232-4a4d-9347-a5f47e64ab17-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:42.531705 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.531685 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4eb96604-4232-4a4d-9347-a5f47e64ab17" (UID: "4eb96604-4232-4a4d-9347-a5f47e64ab17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:22:42.532172 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.532152 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4eb96604-4232-4a4d-9347-a5f47e64ab17" (UID: "4eb96604-4232-4a4d-9347-a5f47e64ab17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:22:42.532172 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.532156 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb96604-4232-4a4d-9347-a5f47e64ab17-kube-api-access-rgtsj" (OuterVolumeSpecName: "kube-api-access-rgtsj") pod "4eb96604-4232-4a4d-9347-a5f47e64ab17" (UID: "4eb96604-4232-4a4d-9347-a5f47e64ab17"). InnerVolumeSpecName "kube-api-access-rgtsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:22:42.631050 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.631019 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:42.631050 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.631043 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eb96604-4232-4a4d-9347-a5f47e64ab17-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:42.631050 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.631052 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtsj\" (UniqueName: \"kubernetes.io/projected/4eb96604-4232-4a4d-9347-a5f47e64ab17-kube-api-access-rgtsj\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:22:42.703373 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.703307 2574 generic.go:356] "Generic (PLEG): container finished" podID="4eb96604-4232-4a4d-9347-a5f47e64ab17" containerID="95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862" exitCode=0 Sep 10 21:22:42.703373 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.703349 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" Sep 10 21:22:42.703373 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.703368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" event={"ID":"4eb96604-4232-4a4d-9347-a5f47e64ab17","Type":"ContainerDied","Data":"95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862"} Sep 10 21:22:42.703599 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.703408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv" event={"ID":"4eb96604-4232-4a4d-9347-a5f47e64ab17","Type":"ContainerDied","Data":"511a0c5f842dac4d5a25511e988216b58d77b02059fa02fb233c2a6030ab8dd9"} Sep 10 21:22:42.703599 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.703428 2574 scope.go:117] "RemoveContainer" containerID="95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862" Sep 10 21:22:42.713054 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.713036 2574 scope.go:117] "RemoveContainer" containerID="95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862" Sep 10 21:22:42.713340 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:22:42.713319 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862\": container with ID starting with 95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862 not found: ID does not exist" containerID="95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862" Sep 10 21:22:42.713402 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.713348 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862"} err="failed to get container status \"95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862\": rpc error: code = NotFound desc = could not find container \"95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862\": container with ID starting with 95227a14094c7ec9c907dc3f447db761a12296e873c7c8169b35fe402b3ba862 not found: ID does not exist" Sep 10 21:22:42.723440 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.723419 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv"] Sep 10 21:22:42.725354 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.725335 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9cce25-679c8dd7f-ttbpv"] Sep 10 21:22:42.805504 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:42.805476 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:44.652326 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:44.652295 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb96604-4232-4a4d-9347-a5f47e64ab17" path="/var/lib/kubelet/pods/4eb96604-4232-4a4d-9347-a5f47e64ab17/volumes" Sep 10 21:22:49.616094 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:49.616060 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:49.712910 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:49.712879 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:49.935007 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:49.934912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:50.388353 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:50.388309 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:22:59.813751 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:22:59.813709 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:00.854767 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:00.854720 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:00.871895 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:00.871859 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw"] Sep 10 21:23:00.872105 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:00.872061 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" podUID="a40bc14c-db86-48fb-ae8c-ecd28f7d3541" containerName="maas-api" containerID="cri-o://cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1" gracePeriod=30 Sep 10 21:23:00.973031 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:00.973000 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:01.085857 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.085822 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:01.134992 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.134969 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:23:01.205580 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.205539 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:01.213650 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.213625 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-metrics-certs\") pod \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " Sep 10 21:23:01.213820 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.213696 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls\") pod \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " Sep 10 21:23:01.213820 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.213739 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-openshift-service-ca\") pod \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " Sep 10 21:23:01.213905 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.213882 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfjd7\" (UniqueName: \"kubernetes.io/projected/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-kube-api-access-rfjd7\") pod \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\" (UID: \"a40bc14c-db86-48fb-ae8c-ecd28f7d3541\") " Sep 10 21:23:01.214136 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.214099 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a40bc14c-db86-48fb-ae8c-ecd28f7d3541" (UID: "a40bc14c-db86-48fb-ae8c-ecd28f7d3541"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:23:01.214312 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.214225 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:23:01.216305 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.216282 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a40bc14c-db86-48fb-ae8c-ecd28f7d3541" (UID: "a40bc14c-db86-48fb-ae8c-ecd28f7d3541"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:23:01.216419 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.216334 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-kube-api-access-rfjd7" (OuterVolumeSpecName: "kube-api-access-rfjd7") pod "a40bc14c-db86-48fb-ae8c-ecd28f7d3541" (UID: "a40bc14c-db86-48fb-ae8c-ecd28f7d3541"). InnerVolumeSpecName "kube-api-access-rfjd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:23:01.216481 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.216419 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a40bc14c-db86-48fb-ae8c-ecd28f7d3541" (UID: "a40bc14c-db86-48fb-ae8c-ecd28f7d3541"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:23:01.315202 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.315159 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:23:01.315202 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.315191 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:23:01.315202 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.315203 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfjd7\" (UniqueName: \"kubernetes.io/projected/a40bc14c-db86-48fb-ae8c-ecd28f7d3541-kube-api-access-rfjd7\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:23:01.328833 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.328765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:01.729477 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.729430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:01.813453 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.813410 2574 generic.go:356] "Generic (PLEG): container finished" podID="a40bc14c-db86-48fb-ae8c-ecd28f7d3541" containerID="cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1" exitCode=0 Sep 10 21:23:01.813597 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.813471 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" Sep 10 21:23:01.813597 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.813474 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" event={"ID":"a40bc14c-db86-48fb-ae8c-ecd28f7d3541","Type":"ContainerDied","Data":"cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1"} Sep 10 21:23:01.813597 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.813591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw" event={"ID":"a40bc14c-db86-48fb-ae8c-ecd28f7d3541","Type":"ContainerDied","Data":"7d5a0a9bacd7b4a04f52eaea2c706636f6285933df3b2e4f486fc70283c3c616"} Sep 10 21:23:01.813725 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.813614 2574 scope.go:117] "RemoveContainer" containerID="cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1" Sep 10 21:23:01.824744 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.824724 2574 scope.go:117] "RemoveContainer" containerID="cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1" Sep 10 21:23:01.825121 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:23:01.825100 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1\": container with ID starting with cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1 not found: ID does not exist" containerID="cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1" Sep 10 21:23:01.825203 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.825134 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1"} err="failed to get container status \"cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1\": rpc error: code = NotFound desc = could not find container \"cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1\": container with ID starting with cf90630dd5d0edd16ae30e64178a32caaab3fafdd1bbbf3d9a4766a60a2645d1 not found: ID does not exist" Sep 10 21:23:01.862607 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.862571 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw"] Sep 10 21:23:01.870103 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.870077 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-95d905-5574c8fdbf-nzvcw"] Sep 10 21:23:01.910400 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:01.910364 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:02.035340 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:02.035305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:02.651041 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:02.650995 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a40bc14c-db86-48fb-ae8c-ecd28f7d3541" path="/var/lib/kubelet/pods/a40bc14c-db86-48fb-ae8c-ecd28f7d3541/volumes" Sep 10 21:23:04.513274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:04.513235 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:04.623219 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:04.623181 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:05.812909 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.812876 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w"] Sep 10 21:23:05.813548 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813528 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84278190-4cf5-45ad-a059-c0539f0506d2" containerName="maas-api" Sep 10 21:23:05.813548 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813549 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84278190-4cf5-45ad-a059-c0539f0506d2" containerName="maas-api" Sep 10 21:23:05.813651 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813567 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb96604-4232-4a4d-9347-a5f47e64ab17" containerName="maas-api" Sep 10 21:23:05.813651 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813572 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb96604-4232-4a4d-9347-a5f47e64ab17" containerName="maas-api" Sep 10 21:23:05.813651 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813582 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40bc14c-db86-48fb-ae8c-ecd28f7d3541" containerName="maas-api" Sep 10 21:23:05.813651 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813587 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40bc14c-db86-48fb-ae8c-ecd28f7d3541" containerName="maas-api" Sep 10 21:23:05.813651 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813595 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cfbe97-0305-4e55-9628-fff0ec78dd1c" containerName="maas-api" Sep 10 21:23:05.813651 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813600 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cfbe97-0305-4e55-9628-fff0ec78dd1c" containerName="maas-api" Sep 10 21:23:05.813885 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813665 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a40bc14c-db86-48fb-ae8c-ecd28f7d3541" containerName="maas-api" Sep 10 21:23:05.813885 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813675 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb96604-4232-4a4d-9347-a5f47e64ab17" containerName="maas-api" Sep 10 21:23:05.813885 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813681 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9cfbe97-0305-4e55-9628-fff0ec78dd1c" containerName="maas-api" Sep 10 21:23:05.813885 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.813687 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84278190-4cf5-45ad-a059-c0539f0506d2" containerName="maas-api" Sep 10 21:23:05.818245 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.818227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.821707 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.821689 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-24b606-serving-cert\"" Sep 10 21:23:05.832634 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.832462 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w"] Sep 10 21:23:05.859666 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.859636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759r5\" (UniqueName: \"kubernetes.io/projected/579dfb44-314d-47b5-88c1-56c1f4e8c135-kube-api-access-759r5\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.859836 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.859683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.859836 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.859713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579dfb44-314d-47b5-88c1-56c1f4e8c135-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.859940 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.859835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-metrics-certs\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.881930 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.881901 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:05.961218 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.961185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759r5\" (UniqueName: \"kubernetes.io/projected/579dfb44-314d-47b5-88c1-56c1f4e8c135-kube-api-access-759r5\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.961424 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.961235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.961424 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.961270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579dfb44-314d-47b5-88c1-56c1f4e8c135-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.961424 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.961381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-metrics-certs\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.961424 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:23:05.961414 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-serving-cert: secret "maas-api-e2e-ipp-a-24b606-serving-cert" not found Sep 10 21:23:05.961648 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:23:05.961507 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls podName:579dfb44-314d-47b5-88c1-56c1f4e8c135 nodeName:}" failed. No retries permitted until 2026-09-10 21:23:06.461484269 +0000 UTC m=+1572.280058370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls") pod "maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" (UID: "579dfb44-314d-47b5-88c1-56c1f4e8c135") : secret "maas-api-e2e-ipp-a-24b606-serving-cert" not found Sep 10 21:23:05.962035 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.962014 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579dfb44-314d-47b5-88c1-56c1f4e8c135-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.963986 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.963965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-metrics-certs\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.971339 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.971314 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759r5\" (UniqueName: \"kubernetes.io/projected/579dfb44-314d-47b5-88c1-56c1f4e8c135-kube-api-access-759r5\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:05.986074 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:05.986021 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:06.374110 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:06.374080 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:06.467107 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:06.467062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:06.469785 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:06.469763 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls\") pod \"maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:06.524307 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:06.524271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:06.625935 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:06.625851 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:06.732109 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:06.732080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:07.025398 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:07.025374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w"] Sep 10 21:23:07.027439 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:23:07.027415 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579dfb44_314d_47b5_88c1_56c1f4e8c135.slice/crio-624b5d962acd65d72b6af6f194e9ee65e281e850d82036cb03cae41f1e94d146 WatchSource:0}: Error finding container 624b5d962acd65d72b6af6f194e9ee65e281e850d82036cb03cae41f1e94d146: Status 404 returned error can't find the container with id 624b5d962acd65d72b6af6f194e9ee65e281e850d82036cb03cae41f1e94d146 Sep 10 21:23:07.853573 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:07.853539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" event={"ID":"579dfb44-314d-47b5-88c1-56c1f4e8c135","Type":"ContainerStarted","Data":"0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8"} Sep 10 21:23:07.853732 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:07.853599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" event={"ID":"579dfb44-314d-47b5-88c1-56c1f4e8c135","Type":"ContainerStarted","Data":"624b5d962acd65d72b6af6f194e9ee65e281e850d82036cb03cae41f1e94d146"} Sep 10 21:23:07.853732 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:07.853672 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:07.905678 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:07.905538 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" podStartSLOduration=2.304128536 podStartE2EDuration="2.90551526s" totalImagesPullingTime="601.386724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:07.899672343 +0000 UTC m=+1573.718246455" watchObservedRunningTime="2026-09-10 21:23:07.90551526 +0000 UTC m=+1573.724089381" Sep 10 21:23:09.510648 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:09.510606 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:09.650547 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:09.650500 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:09.885867 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:09.885799 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:13.864151 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:13.864124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:23:34.292432 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:34.292396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:34.619957 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:34.619862 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:35.262775 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:35.262740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:35.572139 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:35.572107 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:35.819514 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:35.819481 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:38.978538 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:38.978500 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:39.131731 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:39.131698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:40.236019 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.235986 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv"] Sep 10 21:23:40.239888 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.239862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.242437 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.242416 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9f1286-serving-cert\"" Sep 10 21:23:40.251186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.251166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv"] Sep 10 21:23:40.270311 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.270283 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:40.301086 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.301057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d94ef-7608-46a8-880e-10e87d9234f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.301256 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.301111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vrn\" (UniqueName: \"kubernetes.io/projected/f00d94ef-7608-46a8-880e-10e87d9234f0-kube-api-access-f5vrn\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.301256 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.301178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.301398 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.301377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.363792 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.363758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:40.402098 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.402071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.402228 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.402112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d94ef-7608-46a8-880e-10e87d9234f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.402228 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.402134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vrn\" (UniqueName: \"kubernetes.io/projected/f00d94ef-7608-46a8-880e-10e87d9234f0-kube-api-access-f5vrn\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.402228 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.402202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.402743 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.402719 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d94ef-7608-46a8-880e-10e87d9234f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.404745 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.404727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.404825 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.404779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-metrics-certs\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.411675 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.411648 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vrn\" (UniqueName: \"kubernetes.io/projected/f00d94ef-7608-46a8-880e-10e87d9234f0-kube-api-access-f5vrn\") pod \"maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.471591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.471549 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:40.551529 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.551502 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:40.845912 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:40.845828 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv"] Sep 10 21:23:40.851170 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:23:40.851144 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00d94ef_7608_46a8_880e_10e87d9234f0.slice/crio-b80da3f6e1cb9c55416d2ad4fd93671a05f681360279d4e5a261ac9d3031622d WatchSource:0}: Error finding container b80da3f6e1cb9c55416d2ad4fd93671a05f681360279d4e5a261ac9d3031622d: Status 404 returned error can't find the container with id b80da3f6e1cb9c55416d2ad4fd93671a05f681360279d4e5a261ac9d3031622d Sep 10 21:23:41.028733 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:41.028697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" event={"ID":"f00d94ef-7608-46a8-880e-10e87d9234f0","Type":"ContainerStarted","Data":"b80da3f6e1cb9c55416d2ad4fd93671a05f681360279d4e5a261ac9d3031622d"} Sep 10 21:23:41.124676 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:41.124591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:41.295288 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:41.295257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:23:42.035833 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:42.035772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" event={"ID":"f00d94ef-7608-46a8-880e-10e87d9234f0","Type":"ContainerStarted","Data":"ba9a6130598a93099669279b779c228f11d16e27a5ad46a37169e03e0f122951"} Sep 10 21:23:42.036082 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:42.035852 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:42.089455 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:42.089394 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" podStartSLOduration=1.7384359169999999 podStartE2EDuration="2.089377291s" totalImagesPullingTime="350.941374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:42.082874481 +0000 UTC m=+1607.901448595" watchObservedRunningTime="2026-09-10 21:23:42.089377291 +0000 UTC m=+1607.907951411" Sep 10 21:23:48.046908 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:48.046838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:23:54.748439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:23:54.748393 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:03.755247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:03.755210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:08.700714 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:08.700680 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lx7lp_ed07e49a-5e29-4489-93fd-6f96bba473db/payload-processing/0.log" Sep 10 21:24:32.654243 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:32.654201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:54.884635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:54.884598 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lx7lp_ed07e49a-5e29-4489-93fd-6f96bba473db/payload-processing/0.log" Sep 10 21:24:55.298663 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:55.298628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:55.709087 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:55.708990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:56.006178 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:56.006090 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:59.026684 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:59.026649 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:59.118089 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:59.118054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:24:59.492015 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:24:59.491980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:00.190107 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.190073 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb"] Sep 10 21:25:00.194422 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.194399 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.196785 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.196763 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-1ca93e-serving-cert\"" Sep 10 21:25:00.204637 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.204607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb"] Sep 10 21:25:00.222363 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.222323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:00.288577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.288536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdqt\" (UniqueName: \"kubernetes.io/projected/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-kube-api-access-zhdqt\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.288775 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.288594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.288874 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.288784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.288874 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.288864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.316290 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.316254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:00.389681 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.389642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.389944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.389695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.389944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.389718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdqt\" (UniqueName: \"kubernetes.io/projected/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-kube-api-access-zhdqt\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.389944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.389755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.389944 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:25:00.389858 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-serving-cert: secret "maas-api-e2e-ipp-cleanup-1ca93e-serving-cert" not found Sep 10 21:25:00.389944 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:25:00.389941 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls podName:c9978ad9-6e8d-4e08-a6d4-bea5b587700a nodeName:}" failed. No retries permitted until 2026-09-10 21:25:00.889918882 +0000 UTC m=+1686.708492981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" (UID: "c9978ad9-6e8d-4e08-a6d4-bea5b587700a") : secret "maas-api-e2e-ipp-cleanup-1ca93e-serving-cert" not found Sep 10 21:25:00.390491 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.390470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.392446 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.392427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.398674 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.398648 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdqt\" (UniqueName: \"kubernetes.io/projected/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-kube-api-access-zhdqt\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.656623 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.656589 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:00.758622 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.758590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:00.841674 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.841633 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:00.895008 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.894977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:00.897529 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:00.897503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:01.108667 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:01.108619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:01.390639 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:01.390596 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb"] Sep 10 21:25:01.393797 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:25:01.393765 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9978ad9_6e8d_4e08_a6d4_bea5b587700a.slice/crio-b1146ac87b05cddb02b1f91de0acf9e0c2a1ff9cbb591939a834cbfaf4259974 WatchSource:0}: Error finding container b1146ac87b05cddb02b1f91de0acf9e0c2a1ff9cbb591939a834cbfaf4259974: Status 404 returned error can't find the container with id b1146ac87b05cddb02b1f91de0acf9e0c2a1ff9cbb591939a834cbfaf4259974 Sep 10 21:25:01.453002 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:01.452965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" event={"ID":"c9978ad9-6e8d-4e08-a6d4-bea5b587700a","Type":"ContainerStarted","Data":"b1146ac87b05cddb02b1f91de0acf9e0c2a1ff9cbb591939a834cbfaf4259974"} Sep 10 21:25:02.461013 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:02.460973 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" event={"ID":"c9978ad9-6e8d-4e08-a6d4-bea5b587700a","Type":"ContainerStarted","Data":"07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f"} Sep 10 21:25:02.461410 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:02.461071 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:02.480005 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:02.479945 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" podStartSLOduration=2.047034372 podStartE2EDuration="2.47992843s" totalImagesPullingTime="432.894058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:02.478210487 +0000 UTC m=+1688.296784608" watchObservedRunningTime="2026-09-10 21:25:02.47992843 +0000 UTC m=+1688.298502580" Sep 10 21:25:08.472497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:08.472468 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:23.982652 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:23.982590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:30.968564 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:30.968528 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb"] Sep 10 21:25:30.969143 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:30.968791 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" podUID="c9978ad9-6e8d-4e08-a6d4-bea5b587700a" containerName="maas-api" containerID="cri-o://07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f" gracePeriod=30 Sep 10 21:25:31.013422 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.013385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:31.122117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.122083 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:31.221876 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.221765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:31.235710 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.235680 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:31.286424 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.286386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:31.302606 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.302579 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-openshift-service-ca\") pod \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " Sep 10 21:25:31.302746 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.302636 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-metrics-certs\") pod \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " Sep 10 21:25:31.302746 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.302734 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhdqt\" (UniqueName: \"kubernetes.io/projected/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-kube-api-access-zhdqt\") pod \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " Sep 10 21:25:31.302905 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.302760 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls\") pod \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\" (UID: \"c9978ad9-6e8d-4e08-a6d4-bea5b587700a\") " Sep 10 21:25:31.302998 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.302970 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9978ad9-6e8d-4e08-a6d4-bea5b587700a" (UID: "c9978ad9-6e8d-4e08-a6d4-bea5b587700a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:25:31.303122 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.303106 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:31.304917 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.304894 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9978ad9-6e8d-4e08-a6d4-bea5b587700a" (UID: "c9978ad9-6e8d-4e08-a6d4-bea5b587700a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:25:31.305026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.304946 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-kube-api-access-zhdqt" (OuterVolumeSpecName: "kube-api-access-zhdqt") pod "c9978ad9-6e8d-4e08-a6d4-bea5b587700a" (UID: "c9978ad9-6e8d-4e08-a6d4-bea5b587700a"). InnerVolumeSpecName "kube-api-access-zhdqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:25:31.305170 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.305145 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9978ad9-6e8d-4e08-a6d4-bea5b587700a" (UID: "c9978ad9-6e8d-4e08-a6d4-bea5b587700a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:25:31.404632 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.404588 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:31.404632 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.404626 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhdqt\" (UniqueName: \"kubernetes.io/projected/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-kube-api-access-zhdqt\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:31.404632 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.404637 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9978ad9-6e8d-4e08-a6d4-bea5b587700a-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:31.626865 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.626827 2574 generic.go:356] "Generic (PLEG): container finished" podID="c9978ad9-6e8d-4e08-a6d4-bea5b587700a" containerID="07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f" exitCode=0 Sep 10 21:25:31.626865 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.626869 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" Sep 10 21:25:31.627085 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.626909 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" event={"ID":"c9978ad9-6e8d-4e08-a6d4-bea5b587700a","Type":"ContainerDied","Data":"07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f"} Sep 10 21:25:31.627085 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.626960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb" event={"ID":"c9978ad9-6e8d-4e08-a6d4-bea5b587700a","Type":"ContainerDied","Data":"b1146ac87b05cddb02b1f91de0acf9e0c2a1ff9cbb591939a834cbfaf4259974"} Sep 10 21:25:31.627085 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.626982 2574 scope.go:117] "RemoveContainer" containerID="07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f" Sep 10 21:25:31.630632 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.630604 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:31.637018 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.637000 2574 scope.go:117] "RemoveContainer" containerID="07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f" Sep 10 21:25:31.637305 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:25:31.637287 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f\": container with ID starting with 07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f not found: ID does not exist" containerID="07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f" Sep 10 21:25:31.637355 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.637313 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f"} err="failed to get container status \"07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f\": rpc error: code = NotFound desc = could not find container \"07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f\": container with ID starting with 07ab3655cbc783a42b724665e1e0d39163176b6936fb0da739582de6bf98d29f not found: ID does not exist" Sep 10 21:25:31.650715 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.650685 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb"] Sep 10 21:25:31.656137 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:31.656114 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-1ca93e-bfb79f847-vjtqb"] Sep 10 21:25:32.651366 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:32.651335 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9978ad9-6e8d-4e08-a6d4-bea5b587700a" path="/var/lib/kubelet/pods/c9978ad9-6e8d-4e08-a6d4-bea5b587700a/volumes" Sep 10 21:25:37.820462 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:37.820413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:37.921334 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:37.921294 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:38.114465 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:38.114381 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:41.860409 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:41.860358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:52.841658 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:52.841618 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:57.322371 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.322325 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w"] Sep 10 21:25:57.322852 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.322581 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" podUID="579dfb44-314d-47b5-88c1-56c1f4e8c135" containerName="maas-api" containerID="cri-o://0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8" gracePeriod=30 Sep 10 21:25:57.356412 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.356372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:57.501953 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.501924 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:57.593459 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.593429 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:25:57.616179 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.616118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:57.668497 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.668464 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-metrics-certs\") pod \"579dfb44-314d-47b5-88c1-56c1f4e8c135\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " Sep 10 21:25:57.668676 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.668532 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls\") pod \"579dfb44-314d-47b5-88c1-56c1f4e8c135\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " Sep 10 21:25:57.668676 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.668643 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579dfb44-314d-47b5-88c1-56c1f4e8c135-openshift-service-ca\") pod \"579dfb44-314d-47b5-88c1-56c1f4e8c135\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " Sep 10 21:25:57.668676 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.668670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759r5\" (UniqueName: \"kubernetes.io/projected/579dfb44-314d-47b5-88c1-56c1f4e8c135-kube-api-access-759r5\") pod \"579dfb44-314d-47b5-88c1-56c1f4e8c135\" (UID: \"579dfb44-314d-47b5-88c1-56c1f4e8c135\") " Sep 10 21:25:57.669085 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.669058 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579dfb44-314d-47b5-88c1-56c1f4e8c135-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "579dfb44-314d-47b5-88c1-56c1f4e8c135" (UID: "579dfb44-314d-47b5-88c1-56c1f4e8c135"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:25:57.670945 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.670893 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "579dfb44-314d-47b5-88c1-56c1f4e8c135" (UID: "579dfb44-314d-47b5-88c1-56c1f4e8c135"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:25:57.671089 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.670921 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579dfb44-314d-47b5-88c1-56c1f4e8c135-kube-api-access-759r5" (OuterVolumeSpecName: "kube-api-access-759r5") pod "579dfb44-314d-47b5-88c1-56c1f4e8c135" (UID: "579dfb44-314d-47b5-88c1-56c1f4e8c135"). InnerVolumeSpecName "kube-api-access-759r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:25:57.671503 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.671481 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "579dfb44-314d-47b5-88c1-56c1f4e8c135" (UID: "579dfb44-314d-47b5-88c1-56c1f4e8c135"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:25:57.736649 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.736613 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:57.769621 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769591 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579dfb44-314d-47b5-88c1-56c1f4e8c135-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:57.769621 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769622 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759r5\" (UniqueName: \"kubernetes.io/projected/579dfb44-314d-47b5-88c1-56c1f4e8c135-kube-api-access-759r5\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:57.769855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769632 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:57.769855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769640 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579dfb44-314d-47b5-88c1-56c1f4e8c135-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:25:57.769855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769828 2574 generic.go:356] "Generic (PLEG): container finished" podID="579dfb44-314d-47b5-88c1-56c1f4e8c135" containerID="0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8" exitCode=0 Sep 10 21:25:57.770019 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" event={"ID":"579dfb44-314d-47b5-88c1-56c1f4e8c135","Type":"ContainerDied","Data":"0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8"} Sep 10 21:25:57.770019 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769868 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" Sep 10 21:25:57.770019 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w" event={"ID":"579dfb44-314d-47b5-88c1-56c1f4e8c135","Type":"ContainerDied","Data":"624b5d962acd65d72b6af6f194e9ee65e281e850d82036cb03cae41f1e94d146"} Sep 10 21:25:57.770019 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.769948 2574 scope.go:117] "RemoveContainer" containerID="0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8" Sep 10 21:25:57.781011 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.780989 2574 scope.go:117] "RemoveContainer" containerID="0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8" Sep 10 21:25:57.781397 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:25:57.781371 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8\": container with ID starting with 0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8 not found: ID does not exist" containerID="0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8" Sep 10 21:25:57.781488 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.781410 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8"} err="failed to get container status \"0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8\": rpc error: code = NotFound desc = could not find container \"0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8\": container with ID starting with 0449d4ab34f2bc381c784c72593d686adc37aff3d7487c5a77eeea38f420efa8 not found: ID does not exist" Sep 10 21:25:57.798365 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.798328 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w"] Sep 10 21:25:57.801853 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.801821 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-24b606-6657b554bb-klt9w"] Sep 10 21:25:57.818314 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:57.818280 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:58.114481 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:58.114443 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:25:58.651440 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:25:58.651386 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="579dfb44-314d-47b5-88c1-56c1f4e8c135" path="/var/lib/kubelet/pods/579dfb44-314d-47b5-88c1-56c1f4e8c135/volumes" Sep 10 21:26:03.150106 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:03.150062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:21.405699 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:21.405663 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:39.765358 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:39.765326 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:41.561877 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:41.561839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:48.203093 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:48.203001 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:48.269236 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:48.269206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:48.521860 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:48.521757 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:49.194327 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:49.194292 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:26:59.881455 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:59.881403 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv"] Sep 10 21:26:59.881930 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:59.881688 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" podUID="f00d94ef-7608-46a8-880e-10e87d9234f0" containerName="maas-api" containerID="cri-o://ba9a6130598a93099669279b779c228f11d16e27a5ad46a37169e03e0f122951" gracePeriod=30 Sep 10 21:26:59.941772 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:26:59.941736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:00.090068 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.090034 2574 generic.go:356] "Generic (PLEG): container finished" podID="f00d94ef-7608-46a8-880e-10e87d9234f0" containerID="ba9a6130598a93099669279b779c228f11d16e27a5ad46a37169e03e0f122951" exitCode=0 Sep 10 21:27:00.090247 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.090107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" event={"ID":"f00d94ef-7608-46a8-880e-10e87d9234f0","Type":"ContainerDied","Data":"ba9a6130598a93099669279b779c228f11d16e27a5ad46a37169e03e0f122951"} Sep 10 21:27:00.113794 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.113755 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:00.148865 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.148841 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:27:00.261831 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.261779 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-metrics-certs\") pod \"f00d94ef-7608-46a8-880e-10e87d9234f0\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " Sep 10 21:27:00.262033 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.261862 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d94ef-7608-46a8-880e-10e87d9234f0-openshift-service-ca\") pod \"f00d94ef-7608-46a8-880e-10e87d9234f0\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " Sep 10 21:27:00.262033 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.261976 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-maas-api-tls\") pod \"f00d94ef-7608-46a8-880e-10e87d9234f0\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " Sep 10 21:27:00.262033 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.262008 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5vrn\" (UniqueName: \"kubernetes.io/projected/f00d94ef-7608-46a8-880e-10e87d9234f0-kube-api-access-f5vrn\") pod \"f00d94ef-7608-46a8-880e-10e87d9234f0\" (UID: \"f00d94ef-7608-46a8-880e-10e87d9234f0\") " Sep 10 21:27:00.262281 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.262255 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00d94ef-7608-46a8-880e-10e87d9234f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f00d94ef-7608-46a8-880e-10e87d9234f0" (UID: "f00d94ef-7608-46a8-880e-10e87d9234f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:27:00.264237 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.264205 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00d94ef-7608-46a8-880e-10e87d9234f0-kube-api-access-f5vrn" (OuterVolumeSpecName: "kube-api-access-f5vrn") pod "f00d94ef-7608-46a8-880e-10e87d9234f0" (UID: "f00d94ef-7608-46a8-880e-10e87d9234f0"). InnerVolumeSpecName "kube-api-access-f5vrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:27:00.264338 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.264244 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f00d94ef-7608-46a8-880e-10e87d9234f0" (UID: "f00d94ef-7608-46a8-880e-10e87d9234f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:00.264338 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.264234 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f00d94ef-7608-46a8-880e-10e87d9234f0" (UID: "f00d94ef-7608-46a8-880e-10e87d9234f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:00.301505 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.301467 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:00.363022 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.362984 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:00.363022 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.363025 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vrn\" (UniqueName: \"kubernetes.io/projected/f00d94ef-7608-46a8-880e-10e87d9234f0-kube-api-access-f5vrn\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:00.363235 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.363041 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f00d94ef-7608-46a8-880e-10e87d9234f0-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:00.363235 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.363055 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d94ef-7608-46a8-880e-10e87d9234f0-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:00.475480 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.475386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:00.647126 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.647090 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:00.907765 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:00.907724 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:01.096371 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:01.096342 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" Sep 10 21:27:01.096513 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:01.096387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv" event={"ID":"f00d94ef-7608-46a8-880e-10e87d9234f0","Type":"ContainerDied","Data":"b80da3f6e1cb9c55416d2ad4fd93671a05f681360279d4e5a261ac9d3031622d"} Sep 10 21:27:01.096513 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:01.096451 2574 scope.go:117] "RemoveContainer" containerID="ba9a6130598a93099669279b779c228f11d16e27a5ad46a37169e03e0f122951" Sep 10 21:27:01.121636 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:01.121609 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv"] Sep 10 21:27:01.124220 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:01.124198 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9f1286-6dc7d99f7c-8hpgv"] Sep 10 21:27:02.651307 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:02.651266 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00d94ef-7608-46a8-880e-10e87d9234f0" path="/var/lib/kubelet/pods/f00d94ef-7608-46a8-880e-10e87d9234f0/volumes" Sep 10 21:27:07.585665 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:07.585618 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:07.686779 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:07.686740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:07.937635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:07.937559 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:08.544404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:08.544368 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:18.896872 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:18.896838 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v"] Sep 10 21:27:18.897383 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:18.897135 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" podUID="b7e650b6-d3ff-4534-af1e-a9897dc6fd65" containerName="maas-api" containerID="cri-o://dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb" gracePeriod=30 Sep 10 21:27:18.939082 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:18.939047 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:19.062139 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.062103 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:19.156308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.156258 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:27:19.191607 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.191583 2574 generic.go:356] "Generic (PLEG): container finished" podID="b7e650b6-d3ff-4534-af1e-a9897dc6fd65" containerID="dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb" exitCode=0 Sep 10 21:27:19.191739 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.191627 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" Sep 10 21:27:19.191739 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.191671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" event={"ID":"b7e650b6-d3ff-4534-af1e-a9897dc6fd65","Type":"ContainerDied","Data":"dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb"} Sep 10 21:27:19.191739 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.191721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v" event={"ID":"b7e650b6-d3ff-4534-af1e-a9897dc6fd65","Type":"ContainerDied","Data":"eea8179914f95f54ee2cb9a1ff3c29813f90e90e0ed88243e211d48707a9d56a"} Sep 10 21:27:19.191900 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.191751 2574 scope.go:117] "RemoveContainer" containerID="dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb" Sep 10 21:27:19.193186 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.193108 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:19.201888 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.201872 2574 scope.go:117] "RemoveContainer" containerID="dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb" Sep 10 21:27:19.202232 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:27:19.202198 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb\": container with ID starting with dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb not found: ID does not exist" containerID="dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb" Sep 10 21:27:19.202360 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.202240 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb"} err="failed to get container status \"dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb\": rpc error: code = NotFound desc = could not find container \"dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb\": container with ID starting with dc8f07fdf39e7b09f1444389025e724222bf9a85db90821e8747a4f5b053ddbb not found: ID does not exist" Sep 10 21:27:19.245413 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.245384 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-metrics-certs\") pod \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " Sep 10 21:27:19.245601 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.245474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwq8l\" (UniqueName: \"kubernetes.io/projected/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-kube-api-access-wwq8l\") pod \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " Sep 10 21:27:19.245601 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.245565 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls\") pod \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " Sep 10 21:27:19.245729 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.245631 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-openshift-service-ca\") pod \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\" (UID: \"b7e650b6-d3ff-4534-af1e-a9897dc6fd65\") " Sep 10 21:27:19.246058 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.246034 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7e650b6-d3ff-4534-af1e-a9897dc6fd65" (UID: "b7e650b6-d3ff-4534-af1e-a9897dc6fd65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:27:19.247629 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.247600 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7e650b6-d3ff-4534-af1e-a9897dc6fd65" (UID: "b7e650b6-d3ff-4534-af1e-a9897dc6fd65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:19.247712 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.247630 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b7e650b6-d3ff-4534-af1e-a9897dc6fd65" (UID: "b7e650b6-d3ff-4534-af1e-a9897dc6fd65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:19.247712 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.247655 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-kube-api-access-wwq8l" (OuterVolumeSpecName: "kube-api-access-wwq8l") pod "b7e650b6-d3ff-4534-af1e-a9897dc6fd65" (UID: "b7e650b6-d3ff-4534-af1e-a9897dc6fd65"). InnerVolumeSpecName "kube-api-access-wwq8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:27:19.298915 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.298883 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:19.346856 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.346798 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:19.346856 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.346856 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwq8l\" (UniqueName: \"kubernetes.io/projected/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-kube-api-access-wwq8l\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:19.347042 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.346867 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:19.347042 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.346877 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e650b6-d3ff-4534-af1e-a9897dc6fd65-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:19.418423 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.418336 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:19.520982 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.520952 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v"] Sep 10 21:27:19.525524 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.525502 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-af7150-5c54c5d4b9-g577v"] Sep 10 21:27:19.746005 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:19.745923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:20.650160 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:20.650128 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e650b6-d3ff-4534-af1e-a9897dc6fd65" path="/var/lib/kubelet/pods/b7e650b6-d3ff-4534-af1e-a9897dc6fd65/volumes" Sep 10 21:27:27.356292 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:27.356258 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:27.452020 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:27.451978 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:27.711558 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:27.711481 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:28.386307 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:28.386271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:28.821448 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:28.821405 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:34.244368 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:34.244278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:38.350321 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:38.350288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:39.137717 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.137682 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7"] Sep 10 21:27:39.138009 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.137962 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" podUID="4b0a9d2a-45cd-476b-a537-8951622efdeb" containerName="maas-api" containerID="cri-o://3d0a2eae847eeb88c8f451b83c1965c2a748eaed19a72ca86a2c7c0f98036458" gracePeriod=30 Sep 10 21:27:39.237575 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.237538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:39.298257 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.298226 2574 generic.go:356] "Generic (PLEG): container finished" podID="4b0a9d2a-45cd-476b-a537-8951622efdeb" containerID="3d0a2eae847eeb88c8f451b83c1965c2a748eaed19a72ca86a2c7c0f98036458" exitCode=0 Sep 10 21:27:39.298428 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.298345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" event={"ID":"4b0a9d2a-45cd-476b-a537-8951622efdeb","Type":"ContainerDied","Data":"3d0a2eae847eeb88c8f451b83c1965c2a748eaed19a72ca86a2c7c0f98036458"} Sep 10 21:27:39.404218 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.404155 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:27:39.404594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.404387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:39.533131 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.533095 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-metrics-certs\") pod \"4b0a9d2a-45cd-476b-a537-8951622efdeb\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " Sep 10 21:27:39.533313 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.533174 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls\") pod \"4b0a9d2a-45cd-476b-a537-8951622efdeb\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " Sep 10 21:27:39.533313 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.533252 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qggkz\" (UniqueName: \"kubernetes.io/projected/4b0a9d2a-45cd-476b-a537-8951622efdeb-kube-api-access-qggkz\") pod \"4b0a9d2a-45cd-476b-a537-8951622efdeb\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " Sep 10 21:27:39.533439 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.533361 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a9d2a-45cd-476b-a537-8951622efdeb-openshift-service-ca\") pod \"4b0a9d2a-45cd-476b-a537-8951622efdeb\" (UID: \"4b0a9d2a-45cd-476b-a537-8951622efdeb\") " Sep 10 21:27:39.533754 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.533720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0a9d2a-45cd-476b-a537-8951622efdeb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b0a9d2a-45cd-476b-a537-8951622efdeb" (UID: "4b0a9d2a-45cd-476b-a537-8951622efdeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:27:39.535516 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.535492 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b0a9d2a-45cd-476b-a537-8951622efdeb" (UID: "4b0a9d2a-45cd-476b-a537-8951622efdeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:39.536068 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.536043 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b0a9d2a-45cd-476b-a537-8951622efdeb" (UID: "4b0a9d2a-45cd-476b-a537-8951622efdeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:39.536068 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.536053 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0a9d2a-45cd-476b-a537-8951622efdeb-kube-api-access-qggkz" (OuterVolumeSpecName: "kube-api-access-qggkz") pod "4b0a9d2a-45cd-476b-a537-8951622efdeb" (UID: "4b0a9d2a-45cd-476b-a537-8951622efdeb"). InnerVolumeSpecName "kube-api-access-qggkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:27:39.551250 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.551220 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:39.634797 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.634760 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qggkz\" (UniqueName: \"kubernetes.io/projected/4b0a9d2a-45cd-476b-a537-8951622efdeb-kube-api-access-qggkz\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:39.634797 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.634796 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0a9d2a-45cd-476b-a537-8951622efdeb-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:39.634991 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.634829 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:39.634991 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.634843 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0a9d2a-45cd-476b-a537-8951622efdeb-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:39.691366 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.691280 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:39.863304 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:39.863269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:40.261973 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:40.261932 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:40.305604 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:40.305568 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" event={"ID":"4b0a9d2a-45cd-476b-a537-8951622efdeb","Type":"ContainerDied","Data":"eb28d95582724a0d7ba43f107758e3cb7fd0402ba0947aa7512987da075a6497"} Sep 10 21:27:40.305604 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:40.305607 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7" Sep 10 21:27:40.305795 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:40.305672 2574 scope.go:117] "RemoveContainer" containerID="3d0a2eae847eeb88c8f451b83c1965c2a748eaed19a72ca86a2c7c0f98036458" Sep 10 21:27:40.348065 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:40.348035 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7"] Sep 10 21:27:40.353726 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:40.353703 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7810bd-6f4d7db75b-qf4s7"] Sep 10 21:27:40.651185 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:40.651157 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0a9d2a-45cd-476b-a537-8951622efdeb" path="/var/lib/kubelet/pods/4b0a9d2a-45cd-476b-a537-8951622efdeb/volumes" Sep 10 21:27:49.242464 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:49.242426 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:49.502887 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:49.502762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:49.745944 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:49.745895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:27:52.749965 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:52.749924 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-k6nk9"] Sep 10 21:27:52.750553 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:52.750173 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" podUID="f97205e8-2170-49a5-a7e3-293de14a38d2" containerName="manager" containerID="cri-o://e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b" gracePeriod=30 Sep 10 21:27:53.002430 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.002371 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:27:53.168423 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.168399 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-webhook-cert\") pod \"f97205e8-2170-49a5-a7e3-293de14a38d2\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " Sep 10 21:27:53.168596 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.168475 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwmwc\" (UniqueName: \"kubernetes.io/projected/f97205e8-2170-49a5-a7e3-293de14a38d2-kube-api-access-dwmwc\") pod \"f97205e8-2170-49a5-a7e3-293de14a38d2\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " Sep 10 21:27:53.168596 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.168499 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-metrics-certs\") pod \"f97205e8-2170-49a5-a7e3-293de14a38d2\" (UID: \"f97205e8-2170-49a5-a7e3-293de14a38d2\") " Sep 10 21:27:53.170618 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.170585 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f97205e8-2170-49a5-a7e3-293de14a38d2" (UID: "f97205e8-2170-49a5-a7e3-293de14a38d2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:53.170740 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.170653 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f97205e8-2170-49a5-a7e3-293de14a38d2" (UID: "f97205e8-2170-49a5-a7e3-293de14a38d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:27:53.170799 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.170753 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97205e8-2170-49a5-a7e3-293de14a38d2-kube-api-access-dwmwc" (OuterVolumeSpecName: "kube-api-access-dwmwc") pod "f97205e8-2170-49a5-a7e3-293de14a38d2" (UID: "f97205e8-2170-49a5-a7e3-293de14a38d2"). InnerVolumeSpecName "kube-api-access-dwmwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:27:53.270065 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.269986 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-webhook-cert\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:53.270065 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.270016 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmwc\" (UniqueName: \"kubernetes.io/projected/f97205e8-2170-49a5-a7e3-293de14a38d2-kube-api-access-dwmwc\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:53.270065 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.270025 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97205e8-2170-49a5-a7e3-293de14a38d2-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:27:53.374772 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.374738 2574 generic.go:356] "Generic (PLEG): container finished" podID="f97205e8-2170-49a5-a7e3-293de14a38d2" containerID="e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b" exitCode=0 Sep 10 21:27:53.374977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.374792 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" Sep 10 21:27:53.374977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.374848 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" event={"ID":"f97205e8-2170-49a5-a7e3-293de14a38d2","Type":"ContainerDied","Data":"e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b"} Sep 10 21:27:53.374977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.374919 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-k6nk9" event={"ID":"f97205e8-2170-49a5-a7e3-293de14a38d2","Type":"ContainerDied","Data":"792b2b945b1b2aa4c6889d2c4a125c7364a3b3898583a0faa17bcc925370d51b"} Sep 10 21:27:53.374977 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.374948 2574 scope.go:117] "RemoveContainer" containerID="e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b" Sep 10 21:27:53.385078 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.385058 2574 scope.go:117] "RemoveContainer" containerID="e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b" Sep 10 21:27:53.385349 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:27:53.385316 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b\": container with ID starting with e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b not found: ID does not exist" containerID="e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b" Sep 10 21:27:53.385396 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.385358 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b"} err="failed to get container status \"e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b\": rpc error: code = NotFound desc = could not find container \"e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b\": container with ID starting with e03af5707b8b253182f5ac1470c375386d54c859675e21f3ddf4eea238ef520b not found: ID does not exist" Sep 10 21:27:53.438444 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.438417 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-k6nk9"] Sep 10 21:27:53.444142 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:53.444119 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-k6nk9"] Sep 10 21:27:54.650380 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:27:54.650347 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97205e8-2170-49a5-a7e3-293de14a38d2" path="/var/lib/kubelet/pods/f97205e8-2170-49a5-a7e3-293de14a38d2/volumes" Sep 10 21:28:28.326724 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:28.326682 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:28:28.388712 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:28.388678 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:28:54.289933 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.289874 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-zl6jk"] Sep 10 21:28:54.290509 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290488 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e650b6-d3ff-4534-af1e-a9897dc6fd65" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290512 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e650b6-d3ff-4534-af1e-a9897dc6fd65" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290526 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9978ad9-6e8d-4e08-a6d4-bea5b587700a" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290534 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9978ad9-6e8d-4e08-a6d4-bea5b587700a" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290548 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="579dfb44-314d-47b5-88c1-56c1f4e8c135" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290556 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="579dfb44-314d-47b5-88c1-56c1f4e8c135" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290576 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0a9d2a-45cd-476b-a537-8951622efdeb" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290584 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0a9d2a-45cd-476b-a537-8951622efdeb" containerName="maas-api" Sep 10 21:28:54.290594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290592 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00d94ef-7608-46a8-880e-10e87d9234f0" containerName="maas-api" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290601 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00d94ef-7608-46a8-880e-10e87d9234f0" containerName="maas-api" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290625 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97205e8-2170-49a5-a7e3-293de14a38d2" containerName="manager" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290633 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97205e8-2170-49a5-a7e3-293de14a38d2" containerName="manager" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290730 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0a9d2a-45cd-476b-a537-8951622efdeb" containerName="maas-api" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290745 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97205e8-2170-49a5-a7e3-293de14a38d2" containerName="manager" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290756 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9978ad9-6e8d-4e08-a6d4-bea5b587700a" containerName="maas-api" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290764 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00d94ef-7608-46a8-880e-10e87d9234f0" containerName="maas-api" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290774 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="579dfb44-314d-47b5-88c1-56c1f4e8c135" containerName="maas-api" Sep 10 21:28:54.291012 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.290782 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e650b6-d3ff-4534-af1e-a9897dc6fd65" containerName="maas-api" Sep 10 21:28:54.294079 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.294060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.297507 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.297485 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 21:28:54.297635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.297489 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 21:28:54.297635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.297522 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p25h6\"" Sep 10 21:28:54.297635 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.297544 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 21:28:54.304771 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.304748 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-zl6jk"] Sep 10 21:28:54.337664 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.337638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-metrics-certs\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.337777 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.337672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgdh\" (UniqueName: \"kubernetes.io/projected/ed005b94-0a7f-4068-839e-0b5203383e8b-kube-api-access-ffgdh\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.337777 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.337696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-webhook-cert\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.439251 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.439218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-metrics-certs\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.439386 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.439263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgdh\" (UniqueName: \"kubernetes.io/projected/ed005b94-0a7f-4068-839e-0b5203383e8b-kube-api-access-ffgdh\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.439386 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.439296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-webhook-cert\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.441764 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.441738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-metrics-certs\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.441878 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.441766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-webhook-cert\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.447824 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.447790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgdh\" (UniqueName: \"kubernetes.io/projected/ed005b94-0a7f-4068-839e-0b5203383e8b-kube-api-access-ffgdh\") pod \"maas-controller-77cdb86f9b-zl6jk\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.607189 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.607112 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p25h6\"" Sep 10 21:28:54.615043 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.615026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:54.893922 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.893897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-zl6jk"] Sep 10 21:28:54.896461 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:28:54.896433 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded005b94_0a7f_4068_839e_0b5203383e8b.slice/crio-8ca449962f1888802b9a6e8f60fc6807fe194e99b2f1a1723ae3c8cffbd94567 WatchSource:0}: Error finding container 8ca449962f1888802b9a6e8f60fc6807fe194e99b2f1a1723ae3c8cffbd94567: Status 404 returned error can't find the container with id 8ca449962f1888802b9a6e8f60fc6807fe194e99b2f1a1723ae3c8cffbd94567 Sep 10 21:28:54.897884 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:54.897866 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 21:28:55.687765 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:55.687740 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 21:28:55.693018 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:55.692994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" event={"ID":"ed005b94-0a7f-4068-839e-0b5203383e8b","Type":"ContainerStarted","Data":"8ca449962f1888802b9a6e8f60fc6807fe194e99b2f1a1723ae3c8cffbd94567"} Sep 10 21:28:56.699126 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:56.699096 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" event={"ID":"ed005b94-0a7f-4068-839e-0b5203383e8b","Type":"ContainerStarted","Data":"17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb"} Sep 10 21:28:56.699596 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:56.699225 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:28:56.718594 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:28:56.718550 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" podStartSLOduration=1.9315181639999999 podStartE2EDuration="2.718537011s" totalImagesPullingTime="787.018847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:28:56.716871336 +0000 UTC m=+1922.535445470" watchObservedRunningTime="2026-09-10 21:28:56.718537011 +0000 UTC m=+1922.537111131" Sep 10 21:29:07.708182 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:07.708149 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:29:16.239025 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:16.238990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:29:17.060949 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.060908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:29:17.154124 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.154085 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-zl6jk"] Sep 10 21:29:17.154362 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.154337 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" podUID="ed005b94-0a7f-4068-839e-0b5203383e8b" containerName="manager" containerID="cri-o://17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb" gracePeriod=30 Sep 10 21:29:17.405591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.405569 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:29:17.562328 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.562296 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffgdh\" (UniqueName: \"kubernetes.io/projected/ed005b94-0a7f-4068-839e-0b5203383e8b-kube-api-access-ffgdh\") pod \"ed005b94-0a7f-4068-839e-0b5203383e8b\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " Sep 10 21:29:17.562526 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.562398 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-webhook-cert\") pod \"ed005b94-0a7f-4068-839e-0b5203383e8b\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " Sep 10 21:29:17.562526 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.562432 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-metrics-certs\") pod \"ed005b94-0a7f-4068-839e-0b5203383e8b\" (UID: \"ed005b94-0a7f-4068-839e-0b5203383e8b\") " Sep 10 21:29:17.564577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.564541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ed005b94-0a7f-4068-839e-0b5203383e8b" (UID: "ed005b94-0a7f-4068-839e-0b5203383e8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:29:17.564577 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.564560 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed005b94-0a7f-4068-839e-0b5203383e8b-kube-api-access-ffgdh" (OuterVolumeSpecName: "kube-api-access-ffgdh") pod "ed005b94-0a7f-4068-839e-0b5203383e8b" (UID: "ed005b94-0a7f-4068-839e-0b5203383e8b"). InnerVolumeSpecName "kube-api-access-ffgdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:29:17.564729 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.564574 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ed005b94-0a7f-4068-839e-0b5203383e8b" (UID: "ed005b94-0a7f-4068-839e-0b5203383e8b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:29:17.664216 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.664123 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:29:17.664216 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.664155 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgdh\" (UniqueName: \"kubernetes.io/projected/ed005b94-0a7f-4068-839e-0b5203383e8b-kube-api-access-ffgdh\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:29:17.664216 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.664169 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed005b94-0a7f-4068-839e-0b5203383e8b-webhook-cert\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:29:17.808329 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.808290 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed005b94-0a7f-4068-839e-0b5203383e8b" containerID="17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb" exitCode=0 Sep 10 21:29:17.808520 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.808341 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" Sep 10 21:29:17.808520 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.808377 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" event={"ID":"ed005b94-0a7f-4068-839e-0b5203383e8b","Type":"ContainerDied","Data":"17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb"} Sep 10 21:29:17.808520 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.808438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-zl6jk" event={"ID":"ed005b94-0a7f-4068-839e-0b5203383e8b","Type":"ContainerDied","Data":"8ca449962f1888802b9a6e8f60fc6807fe194e99b2f1a1723ae3c8cffbd94567"} Sep 10 21:29:17.808520 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.808462 2574 scope.go:117] "RemoveContainer" containerID="17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb" Sep 10 21:29:17.822078 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.821951 2574 scope.go:117] "RemoveContainer" containerID="17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb" Sep 10 21:29:17.822681 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:29:17.822645 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb\": container with ID starting with 17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb not found: ID does not exist" containerID="17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb" Sep 10 21:29:17.822829 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.822684 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb"} err="failed to get container status \"17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb\": rpc error: code = NotFound desc = could not find container \"17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb\": container with ID starting with 17762d88a97ce523039a31cb553b7e5bedf361468bb65a4882ea01eb3e46abeb not found: ID does not exist" Sep 10 21:29:17.835778 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.835742 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-zl6jk"] Sep 10 21:29:17.839889 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:17.839864 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-zl6jk"] Sep 10 21:29:18.550526 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.550481 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:29:18.651739 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.651705 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed005b94-0a7f-4068-839e-0b5203383e8b" path="/var/lib/kubelet/pods/ed005b94-0a7f-4068-839e-0b5203383e8b/volumes" Sep 10 21:29:18.672669 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.672635 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-t7m8f"] Sep 10 21:29:18.673139 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.673121 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed005b94-0a7f-4068-839e-0b5203383e8b" containerName="manager" Sep 10 21:29:18.673236 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.673141 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed005b94-0a7f-4068-839e-0b5203383e8b" containerName="manager" Sep 10 21:29:18.673295 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.673238 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed005b94-0a7f-4068-839e-0b5203383e8b" containerName="manager" Sep 10 21:29:18.677791 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.677771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.681175 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.681156 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 21:29:18.681175 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.681164 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 21:29:18.681349 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.681157 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p25h6\"" Sep 10 21:29:18.681349 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.681211 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 21:29:18.687636 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.687613 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-t7m8f"] Sep 10 21:29:18.776014 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.775979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-webhook-cert\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.776166 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.776040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9x5\" (UniqueName: \"kubernetes.io/projected/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-kube-api-access-8r9x5\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.776226 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.776203 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-metrics-certs\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.877437 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.877359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-webhook-cert\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.877437 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.877409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9x5\" (UniqueName: \"kubernetes.io/projected/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-kube-api-access-8r9x5\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.877620 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.877478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-metrics-certs\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.879989 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.879966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-webhook-cert\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.880111 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.880000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-metrics-certs\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.886077 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.886056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9x5\" (UniqueName: \"kubernetes.io/projected/f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f-kube-api-access-8r9x5\") pod \"maas-controller-77cdb86f9b-t7m8f\" (UID: \"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f\") " pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:18.988929 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:18.988901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:19.262338 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:19.262253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77cdb86f9b-t7m8f"] Sep 10 21:29:19.264788 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:29:19.264760 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f270dc_16ab_4ff7_9fd0_64b7e4e8421f.slice/crio-074d7bbf6783f8490ccdf639b88b10b2d1b9f7efcd3708e03dabde0f9daa5f9c WatchSource:0}: Error finding container 074d7bbf6783f8490ccdf639b88b10b2d1b9f7efcd3708e03dabde0f9daa5f9c: Status 404 returned error can't find the container with id 074d7bbf6783f8490ccdf639b88b10b2d1b9f7efcd3708e03dabde0f9daa5f9c Sep 10 21:29:19.821685 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:19.821602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" event={"ID":"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f","Type":"ContainerStarted","Data":"acd7dbbe52440fcb7f4b6702cb571b95f52cdbff8165f9e69d73b7c80ca03ee1"} Sep 10 21:29:19.821685 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:19.821658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" event={"ID":"f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f","Type":"ContainerStarted","Data":"074d7bbf6783f8490ccdf639b88b10b2d1b9f7efcd3708e03dabde0f9daa5f9c"} Sep 10 21:29:19.822101 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:19.821691 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:29:19.843695 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:19.843637 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" podStartSLOduration=1.556540682 podStartE2EDuration="1.843618437s" totalImagesPullingTime="287.077755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:29:19.841382876 +0000 UTC m=+1945.659956996" watchObservedRunningTime="2026-09-10 21:29:19.843618437 +0000 UTC m=+1945.662192559" Sep 10 21:29:30.830257 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:29:30.830222 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77cdb86f9b-t7m8f" Sep 10 21:30:26.743870 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:30:26.743830 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:30:26.806250 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:30:26.806215 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:32:08.943157 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:08.943103 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:32:15.958884 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:15.958844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:32:32.579323 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:32.579286 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:32:34.295980 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:34.295947 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:32:43.862213 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:43.862175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:32:58.021471 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.021381 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g"] Sep 10 21:32:58.024976 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.024949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.033934 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.033909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g"] Sep 10 21:32:58.069872 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.069842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f85e7-0512-478f-ab94-d606c82e4f7d-maas-api-tls\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.070024 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.069900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f85e7-0512-478f-ab94-d606c82e4f7d-metrics-certs\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.070024 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.069945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbh9\" (UniqueName: \"kubernetes.io/projected/d38f85e7-0512-478f-ab94-d606c82e4f7d-kube-api-access-xlbh9\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.070117 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.070015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f85e7-0512-478f-ab94-d606c82e4f7d-openshift-service-ca\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.171414 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.171380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f85e7-0512-478f-ab94-d606c82e4f7d-maas-api-tls\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.171583 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.171431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f85e7-0512-478f-ab94-d606c82e4f7d-metrics-certs\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.171583 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.171473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbh9\" (UniqueName: \"kubernetes.io/projected/d38f85e7-0512-478f-ab94-d606c82e4f7d-kube-api-access-xlbh9\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.171583 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.171511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f85e7-0512-478f-ab94-d606c82e4f7d-openshift-service-ca\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.172225 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.172204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38f85e7-0512-478f-ab94-d606c82e4f7d-openshift-service-ca\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.174130 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.174105 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38f85e7-0512-478f-ab94-d606c82e4f7d-metrics-certs\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.174244 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.174227 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d38f85e7-0512-478f-ab94-d606c82e4f7d-maas-api-tls\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.179424 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.179396 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbh9\" (UniqueName: \"kubernetes.io/projected/d38f85e7-0512-478f-ab94-d606c82e4f7d-kube-api-access-xlbh9\") pod \"maas-api-84ff5476d-ghk5g\" (UID: \"d38f85e7-0512-478f-ab94-d606c82e4f7d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.336777 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.336742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:58.616426 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.616399 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g"] Sep 10 21:32:58.618730 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:32:58.618703 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38f85e7_0512_478f_ab94_d606c82e4f7d.slice/crio-5d78c68191a3750e3779ba58fd391f41430c495c7943d7598fa3a1574c2d17b4 WatchSource:0}: Error finding container 5d78c68191a3750e3779ba58fd391f41430c495c7943d7598fa3a1574c2d17b4: Status 404 returned error can't find the container with id 5d78c68191a3750e3779ba58fd391f41430c495c7943d7598fa3a1574c2d17b4 Sep 10 21:32:58.938561 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:58.938475 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" event={"ID":"d38f85e7-0512-478f-ab94-d606c82e4f7d","Type":"ContainerStarted","Data":"5d78c68191a3750e3779ba58fd391f41430c495c7943d7598fa3a1574c2d17b4"} Sep 10 21:32:59.947893 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:59.947850 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" event={"ID":"d38f85e7-0512-478f-ab94-d606c82e4f7d","Type":"ContainerStarted","Data":"a79901beb64c6f0514185d1bb536fbd32e9d82af02fe8ba82d9739f45f058f76"} Sep 10 21:32:59.948240 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:59.947923 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:32:59.964258 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:32:59.964198 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" podStartSLOduration=1.5761333990000002 podStartE2EDuration="1.964178501s" totalImagesPullingTime="388.045102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:59.96356456 +0000 UTC m=+2165.782138683" watchObservedRunningTime="2026-09-10 21:32:59.964178501 +0000 UTC m=+2165.782752623" Sep 10 21:33:05.958997 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:05.958963 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84ff5476d-ghk5g" Sep 10 21:33:15.566308 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:15.566270 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq"] Sep 10 21:33:15.566855 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:15.566550 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" podUID="a3dacc6d-1984-42c4-9274-5a1fbf64d090" containerName="maas-api" containerID="cri-o://7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d" gracePeriod=30 Sep 10 21:33:16.820300 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.820279 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:33:16.954992 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.954917 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dacc6d-1984-42c4-9274-5a1fbf64d090-openshift-service-ca\") pod \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " Sep 10 21:33:16.955129 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.955036 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8wwm\" (UniqueName: \"kubernetes.io/projected/a3dacc6d-1984-42c4-9274-5a1fbf64d090-kube-api-access-v8wwm\") pod \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " Sep 10 21:33:16.955129 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.955090 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-maas-api-tls\") pod \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " Sep 10 21:33:16.955129 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.955115 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-metrics-certs\") pod \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\" (UID: \"a3dacc6d-1984-42c4-9274-5a1fbf64d090\") " Sep 10 21:33:16.955327 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.955304 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dacc6d-1984-42c4-9274-5a1fbf64d090-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3dacc6d-1984-42c4-9274-5a1fbf64d090" (UID: "a3dacc6d-1984-42c4-9274-5a1fbf64d090"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 21:33:16.955453 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.955438 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3dacc6d-1984-42c4-9274-5a1fbf64d090-openshift-service-ca\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:33:16.957265 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.957239 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3dacc6d-1984-42c4-9274-5a1fbf64d090" (UID: "a3dacc6d-1984-42c4-9274-5a1fbf64d090"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:33:16.957361 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.957301 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a3dacc6d-1984-42c4-9274-5a1fbf64d090" (UID: "a3dacc6d-1984-42c4-9274-5a1fbf64d090"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 21:33:16.957429 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:16.957398 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dacc6d-1984-42c4-9274-5a1fbf64d090-kube-api-access-v8wwm" (OuterVolumeSpecName: "kube-api-access-v8wwm") pod "a3dacc6d-1984-42c4-9274-5a1fbf64d090" (UID: "a3dacc6d-1984-42c4-9274-5a1fbf64d090"). InnerVolumeSpecName "kube-api-access-v8wwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 21:33:17.040859 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.040793 2574 generic.go:356] "Generic (PLEG): container finished" podID="a3dacc6d-1984-42c4-9274-5a1fbf64d090" containerID="7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d" exitCode=0 Sep 10 21:33:17.041009 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.040860 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" Sep 10 21:33:17.041009 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.040879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" event={"ID":"a3dacc6d-1984-42c4-9274-5a1fbf64d090","Type":"ContainerDied","Data":"7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d"} Sep 10 21:33:17.041009 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.040935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq" event={"ID":"a3dacc6d-1984-42c4-9274-5a1fbf64d090","Type":"ContainerDied","Data":"06b3f75770fe22119b151e4c842733243990b65db64a93e1bedd5a5d6166f9ed"} Sep 10 21:33:17.041009 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.040962 2574 scope.go:117] "RemoveContainer" containerID="7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d" Sep 10 21:33:17.050127 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.050108 2574 scope.go:117] "RemoveContainer" containerID="7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d" Sep 10 21:33:17.050385 ip-10-0-132-255 kubenswrapper[2574]: E0910 21:33:17.050365 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d\": container with ID starting with 7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d not found: ID does not exist" containerID="7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d" Sep 10 21:33:17.050461 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.050391 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d"} err="failed to get container status \"7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d\": rpc error: code = NotFound desc = could not find container \"7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d\": container with ID starting with 7610aa92fe997c15fe629d75e3c6fea2b97ccda90308f653e1e04d533ad79d0d not found: ID does not exist" Sep 10 21:33:17.056219 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.056202 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8wwm\" (UniqueName: \"kubernetes.io/projected/a3dacc6d-1984-42c4-9274-5a1fbf64d090-kube-api-access-v8wwm\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:33:17.056277 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.056222 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-maas-api-tls\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:33:17.056277 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.056231 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3dacc6d-1984-42c4-9274-5a1fbf64d090-metrics-certs\") on node \"ip-10-0-132-255.ec2.internal\" DevicePath \"\"" Sep 10 21:33:17.063782 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.063758 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq"] Sep 10 21:33:17.068420 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:17.068399 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b78667ccb-dtkzq"] Sep 10 21:33:18.651535 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:18.651502 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3dacc6d-1984-42c4-9274-5a1fbf64d090" path="/var/lib/kubelet/pods/a3dacc6d-1984-42c4-9274-5a1fbf64d090/volumes" Sep 10 21:33:33.910150 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:33.910121 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5qckg_8cc26d22-85b6-4d2e-a046-f665fc4fb3bb/manager/0.log" Sep 10 21:33:34.028285 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:34.028254 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zd8bl_e11d85cb-a02f-4982-a9b8-a69462a98e71/manager/0.log" Sep 10 21:33:34.147982 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:34.147946 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77cdb86f9b-t7m8f_f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f/manager/0.log" Sep 10 21:33:34.267841 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:34.267744 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-txpqf_7c189779-ccac-40cb-87fb-01dd5dd261c4/server/0.log" Sep 10 21:33:34.384048 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:34.384016 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jgtss_91fa8523-3c84-41ba-a873-5b1fe5bf3f68/manager/0.log" Sep 10 21:33:34.769192 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:34.769163 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-rbzpt_0ca938d6-275e-41dc-9815-2577f9101c39/copy-manifests/0.log" Sep 10 21:33:34.782570 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:34.782542 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-rbzpt_0ca938d6-275e-41dc-9815-2577f9101c39/manager/0.log" Sep 10 21:33:35.134854 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:35.134826 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84ff5476d-ghk5g_d38f85e7-0512-478f-ab94-d606c82e4f7d/maas-api/0.log" Sep 10 21:33:35.590032 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:35.590004 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8rnk7_f06121b9-4285-4b6d-a17e-92862bdbc7bc/postgres/0.log" Sep 10 21:33:36.632299 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:36.627535 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-lh6qp_f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018/manager/0.log" Sep 10 21:33:37.479293 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:37.479266 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc_61512794-0fb8-4fa7-be7e-50c6a850b78e/istio-proxy/0.log" Sep 10 21:33:37.949668 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:37.949625 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lf76f_58a2ca1c-58c8-445f-a151-3875cc3b4421/istio-proxy/1.log" Sep 10 21:33:38.073315 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:38.073282 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nbkbj_9c873f35-55af-4bd2-af81-f34a6d48fc23/payload-pre-processing/0.log" Sep 10 21:33:38.194064 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:38.194038 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lx7lp_ed07e49a-5e29-4489-93fd-6f96bba473db/payload-processing/0.log" Sep 10 21:33:38.681029 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:38.680994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-48x77_7aa718eb-9e98-4588-8f23-957544bce209/main/0.log" Sep 10 21:33:38.926876 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:38.926847 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-pfkrm_233b67a1-165f-47c5-bf7a-636b0047bc91/main/0.log" Sep 10 21:33:39.282972 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:39.282945 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-46xp8_90bb4c10-1396-4d7d-9066-e713d9d96210/main/0.log" Sep 10 21:33:45.549719 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:45.549680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:33:45.650298 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:45.650248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" limit=1024 Sep 10 21:33:48.440137 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:48.440111 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jsfgg_8c4c92c2-36db-4308-b268-f01026044047/global-pull-secret-syncer/0.log" Sep 10 21:33:48.539638 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:48.539601 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gfcs9_1bedad80-3589-4b87-a28a-47927ae56347/konnectivity-agent/0.log" Sep 10 21:33:48.656133 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:48.656108 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-255.ec2.internal_0f953879e24b63da8b00052cd2968abb/haproxy/0.log" Sep 10 21:33:52.828373 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:52.828333 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-lh6qp_f5d8fe45-2d2e-44a9-bb6a-bf3c2127f018/manager/0.log" Sep 10 21:33:54.758623 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:54.758590 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-djpln_fe947841-41c5-485b-8061-72984004c125/cluster-monitoring-operator/0.log" Sep 10 21:33:54.785193 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:54.785169 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-m7djv_2a74ba8c-089a-40a5-8321-da76e624fe3d/kube-state-metrics/0.log" Sep 10 21:33:54.803333 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:54.803274 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-m7djv_2a74ba8c-089a-40a5-8321-da76e624fe3d/kube-rbac-proxy-main/0.log" Sep 10 21:33:54.824088 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:54.824064 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-m7djv_2a74ba8c-089a-40a5-8321-da76e624fe3d/kube-rbac-proxy-self/0.log" Sep 10 21:33:54.851505 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:54.851475 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b858df774-xzbxq_a00d1f1b-56de-47df-8413-26a7d4e9cf05/metrics-server/0.log" Sep 10 21:33:55.091042 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.090958 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j8cbq_37091e43-3957-4338-90a8-8c51e7f709ee/node-exporter/0.log" Sep 10 21:33:55.113323 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.113298 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j8cbq_37091e43-3957-4338-90a8-8c51e7f709ee/kube-rbac-proxy/0.log" Sep 10 21:33:55.138397 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.138379 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j8cbq_37091e43-3957-4338-90a8-8c51e7f709ee/init-textfile/0.log" Sep 10 21:33:55.257141 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.257116 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d173090-cb95-425d-bb96-16daa5a9f73f/prometheus/0.log" Sep 10 21:33:55.282889 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.282866 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d173090-cb95-425d-bb96-16daa5a9f73f/config-reloader/0.log" Sep 10 21:33:55.304345 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.304324 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d173090-cb95-425d-bb96-16daa5a9f73f/thanos-sidecar/0.log" Sep 10 21:33:55.330963 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.330945 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d173090-cb95-425d-bb96-16daa5a9f73f/kube-rbac-proxy-web/0.log" Sep 10 21:33:55.352103 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.352043 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d173090-cb95-425d-bb96-16daa5a9f73f/kube-rbac-proxy/0.log" Sep 10 21:33:55.374773 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.374752 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d173090-cb95-425d-bb96-16daa5a9f73f/kube-rbac-proxy-thanos/0.log" Sep 10 21:33:55.404384 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.404337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d173090-cb95-425d-bb96-16daa5a9f73f/init-config-reloader/0.log" Sep 10 21:33:55.433549 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.433532 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-84qt2_3f6ca02d-da18-4fa9-b783-7b0bbed28928/prometheus-operator/0.log" Sep 10 21:33:55.458404 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.458380 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-84qt2_3f6ca02d-da18-4fa9-b783-7b0bbed28928/kube-rbac-proxy/0.log" Sep 10 21:33:55.497953 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.497928 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-pnqrj_655c6939-cce7-4971-ae37-3cd37d2b628f/prometheus-operator-admission-webhook/0.log" Sep 10 21:33:55.731517 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.731437 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d5976855f-ktlsq_e80058f1-e403-4771-b1ff-0d0206ba1f1a/thanos-query/0.log" Sep 10 21:33:55.758853 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.758801 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d5976855f-ktlsq_e80058f1-e403-4771-b1ff-0d0206ba1f1a/kube-rbac-proxy-web/0.log" Sep 10 21:33:55.793761 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.793732 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d5976855f-ktlsq_e80058f1-e403-4771-b1ff-0d0206ba1f1a/kube-rbac-proxy/0.log" Sep 10 21:33:55.827650 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.827564 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d5976855f-ktlsq_e80058f1-e403-4771-b1ff-0d0206ba1f1a/prom-label-proxy/0.log" Sep 10 21:33:55.851416 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.851388 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d5976855f-ktlsq_e80058f1-e403-4771-b1ff-0d0206ba1f1a/kube-rbac-proxy-rules/0.log" Sep 10 21:33:55.875692 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:55.875663 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d5976855f-ktlsq_e80058f1-e403-4771-b1ff-0d0206ba1f1a/kube-rbac-proxy-metrics/0.log" Sep 10 21:33:57.380351 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:57.380319 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-l5wpg_18d9a069-cb74-40d1-ac8a-e774358f015a/networking-console-plugin/0.log" Sep 10 21:33:58.747682 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.747653 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5"] Sep 10 21:33:58.748161 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.748120 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3dacc6d-1984-42c4-9274-5a1fbf64d090" containerName="maas-api" Sep 10 21:33:58.748161 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.748133 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3dacc6d-1984-42c4-9274-5a1fbf64d090" containerName="maas-api" Sep 10 21:33:58.748232 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.748207 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3dacc6d-1984-42c4-9274-5a1fbf64d090" containerName="maas-api" Sep 10 21:33:58.752591 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.752572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.755265 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.755246 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hcpt\"/\"openshift-service-ca.crt\"" Sep 10 21:33:58.756112 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.756095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hcpt\"/\"default-dockercfg-w8z8c\"" Sep 10 21:33:58.756199 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.756130 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hcpt\"/\"kube-root-ca.crt\"" Sep 10 21:33:58.764603 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.764580 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5"] Sep 10 21:33:58.835442 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.835407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-lib-modules\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.835572 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.835445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stz2m\" (UniqueName: \"kubernetes.io/projected/0d13d0e5-8528-4ce5-b542-a560e02502dd-kube-api-access-stz2m\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.835572 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.835473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-proc\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.835572 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.835548 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-podres\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.835683 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.835623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-sys\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.936653 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-lib-modules\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.936799 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stz2m\" (UniqueName: \"kubernetes.io/projected/0d13d0e5-8528-4ce5-b542-a560e02502dd-kube-api-access-stz2m\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.936799 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-proc\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.936799 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-podres\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.936799 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-lib-modules\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.937026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-proc\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.937026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-sys\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.937026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-sys\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.937026 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.936936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d13d0e5-8528-4ce5-b542-a560e02502dd-podres\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:58.945766 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:58.945748 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stz2m\" (UniqueName: \"kubernetes.io/projected/0d13d0e5-8528-4ce5-b542-a560e02502dd-kube-api-access-stz2m\") pod \"perf-node-gather-daemonset-7gqw5\" (UID: \"0d13d0e5-8528-4ce5-b542-a560e02502dd\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:59.063198 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:59.063169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:33:59.335960 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:33:59.335935 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5"] Sep 10 21:33:59.337962 ip-10-0-132-255 kubenswrapper[2574]: W0910 21:33:59.337937 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d13d0e5_8528_4ce5_b542_a560e02502dd.slice/crio-124f8b6849071b50c9a1e7ed481292e22bcb4320c4c32b6d7d33d8f3b0fddd5f WatchSource:0}: Error finding container 124f8b6849071b50c9a1e7ed481292e22bcb4320c4c32b6d7d33d8f3b0fddd5f: Status 404 returned error can't find the container with id 124f8b6849071b50c9a1e7ed481292e22bcb4320c4c32b6d7d33d8f3b0fddd5f Sep 10 21:34:00.263450 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:00.263415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" event={"ID":"0d13d0e5-8528-4ce5-b542-a560e02502dd","Type":"ContainerStarted","Data":"3c92e776495c8e2797370e7b1f0d50c8c0f653392b0ed8edb6334d956b1e9dd6"} Sep 10 21:34:00.263879 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:00.263473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" event={"ID":"0d13d0e5-8528-4ce5-b542-a560e02502dd","Type":"ContainerStarted","Data":"124f8b6849071b50c9a1e7ed481292e22bcb4320c4c32b6d7d33d8f3b0fddd5f"} Sep 10 21:34:00.263879 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:00.263512 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:34:00.283125 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:00.283068 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" podStartSLOduration=2.283053066 podStartE2EDuration="2.283053066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:34:00.280522004 +0000 UTC m=+2226.099096125" watchObservedRunningTime="2026-09-10 21:34:00.283053066 +0000 UTC m=+2226.101627426" Sep 10 21:34:00.639689 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:00.639661 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbkpz_597fc486-2ab7-49cb-91ae-9d4a37bd174b/dns/0.log" Sep 10 21:34:00.661481 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:00.661460 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbkpz_597fc486-2ab7-49cb-91ae-9d4a37bd174b/kube-rbac-proxy/0.log" Sep 10 21:34:00.713182 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:00.713154 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tc7ff_b7b16656-f7d0-4f2e-9326-fa2dda31ff23/dns-node-resolver/0.log" Sep 10 21:34:01.355857 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:01.355828 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pmgvp_b62fe6e0-e353-446e-af6f-89c408244412/node-ca/0.log" Sep 10 21:34:02.297094 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:02.297065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc_61512794-0fb8-4fa7-be7e-50c6a850b78e/istio-proxy/0.log" Sep 10 21:34:02.593337 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:02.593256 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lf76f_58a2ca1c-58c8-445f-a151-3875cc3b4421/istio-proxy/0.log" Sep 10 21:34:02.600846 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:02.600827 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lf76f_58a2ca1c-58c8-445f-a151-3875cc3b4421/istio-proxy/1.log" Sep 10 21:34:02.633271 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:02.633243 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nbkbj_9c873f35-55af-4bd2-af81-f34a6d48fc23/payload-pre-processing/0.log" Sep 10 21:34:02.683983 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:02.683954 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lx7lp_ed07e49a-5e29-4489-93fd-6f96bba473db/payload-processing/0.log" Sep 10 21:34:03.612791 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:03.612763 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rzdqm_a91b9b2b-9609-4f4d-9bcd-07e184a8f19d/serve-healthcheck-canary/0.log" Sep 10 21:34:04.088375 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:04.088348 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-289rz_37516ba4-a1d3-41a7-b5b3-5e4ab1339957/kube-rbac-proxy/0.log" Sep 10 21:34:04.114385 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:04.114356 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-289rz_37516ba4-a1d3-41a7-b5b3-5e4ab1339957/exporter/0.log" Sep 10 21:34:04.134223 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:04.134201 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-289rz_37516ba4-a1d3-41a7-b5b3-5e4ab1339957/extractor/0.log" Sep 10 21:34:06.278232 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.278207 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-7gqw5" Sep 10 21:34:06.420860 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.420832 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5qckg_8cc26d22-85b6-4d2e-a046-f665fc4fb3bb/manager/0.log" Sep 10 21:34:06.585678 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.585598 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zd8bl_e11d85cb-a02f-4982-a9b8-a69462a98e71/manager/0.log" Sep 10 21:34:06.634632 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.634605 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77cdb86f9b-t7m8f_f8f270dc-16ab-4ff7-9fd0-64b7e4e8421f/manager/0.log" Sep 10 21:34:06.653522 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.653501 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-txpqf_7c189779-ccac-40cb-87fb-01dd5dd261c4/server/0.log" Sep 10 21:34:06.732274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.732251 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jgtss_91fa8523-3c84-41ba-a873-5b1fe5bf3f68/manager/0.log" Sep 10 21:34:06.907562 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.907475 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-rbzpt_0ca938d6-275e-41dc-9815-2577f9101c39/manager/0.log" Sep 10 21:34:06.923735 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:06.923714 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-rbzpt_0ca938d6-275e-41dc-9815-2577f9101c39/copy-manifests/0.log" Sep 10 21:34:08.360357 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:08.360333 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-799dfcfb97-kpph9_6c66153f-4ac5-4304-ac05-b3f32173e600/manager/0.log" Sep 10 21:34:08.390485 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:08.390456 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lh2vj_391f4f10-e58b-405a-b237-7e61b93d1d02/openshift-lws-operator/0.log" Sep 10 21:34:08.822499 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:08.822464 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc_61512794-0fb8-4fa7-be7e-50c6a850b78e/istio-proxy/0.log" Sep 10 21:34:09.083486 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:09.083461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lf76f_58a2ca1c-58c8-445f-a151-3875cc3b4421/istio-proxy/0.log" Sep 10 21:34:09.091889 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:09.091865 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lf76f_58a2ca1c-58c8-445f-a151-3875cc3b4421/istio-proxy/1.log" Sep 10 21:34:09.114361 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:09.114340 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nbkbj_9c873f35-55af-4bd2-af81-f34a6d48fc23/payload-pre-processing/0.log" Sep 10 21:34:09.137516 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:09.137473 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lx7lp_ed07e49a-5e29-4489-93fd-6f96bba473db/payload-processing/0.log" Sep 10 21:34:10.637642 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:10.637618 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-frqkc_61512794-0fb8-4fa7-be7e-50c6a850b78e/istio-proxy/0.log" Sep 10 21:34:10.899274 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:10.899234 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lf76f_58a2ca1c-58c8-445f-a151-3875cc3b4421/istio-proxy/0.log" Sep 10 21:34:10.906858 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:10.906835 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lf76f_58a2ca1c-58c8-445f-a151-3875cc3b4421/istio-proxy/1.log" Sep 10 21:34:10.935075 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:10.935049 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-nbkbj_9c873f35-55af-4bd2-af81-f34a6d48fc23/payload-pre-processing/0.log" Sep 10 21:34:10.962727 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:10.962703 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-lx7lp_ed07e49a-5e29-4489-93fd-6f96bba473db/payload-processing/0.log" Sep 10 21:34:15.000649 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.000623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82h58_544862f7-a4c6-4753-9932-22668843f699/kube-multus-additional-cni-plugins/0.log" Sep 10 21:34:15.022367 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.022334 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82h58_544862f7-a4c6-4753-9932-22668843f699/egress-router-binary-copy/0.log" Sep 10 21:34:15.042232 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.042210 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82h58_544862f7-a4c6-4753-9932-22668843f699/cni-plugins/0.log" Sep 10 21:34:15.062056 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.062033 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82h58_544862f7-a4c6-4753-9932-22668843f699/bond-cni-plugin/0.log" Sep 10 21:34:15.084746 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.084726 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82h58_544862f7-a4c6-4753-9932-22668843f699/routeoverride-cni/0.log" Sep 10 21:34:15.104886 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.104867 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82h58_544862f7-a4c6-4753-9932-22668843f699/whereabouts-cni-bincopy/0.log" Sep 10 21:34:15.124746 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.124728 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-82h58_544862f7-a4c6-4753-9932-22668843f699/whereabouts-cni/0.log" Sep 10 21:34:15.569463 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.569426 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t57dn_9ca6eb1f-39ca-4e13-9451-c2e2ac879135/kube-multus/0.log" Sep 10 21:34:15.589951 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.589926 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kflkv_d9cb1d07-086d-48a2-92a7-f8061493cead/network-metrics-daemon/0.log" Sep 10 21:34:15.608589 ip-10-0-132-255 kubenswrapper[2574]: I0910 21:34:15.608568 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kflkv_d9cb1d07-086d-48a2-92a7-f8061493cead/kube-rbac-proxy/0.log"