Sep 08 00:32:44.390965 ip-10-0-142-98 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 00:32:44.824659 ip-10-0-142-98 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 00:32:44.824659 ip-10-0-142-98 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 00:32:44.824659 ip-10-0-142-98 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 00:32:44.824659 ip-10-0-142-98 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 00:32:44.824659 ip-10-0-142-98 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 00:32:44.826726 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.826644    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 00:32:44.830865 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830850    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:32:44.830865 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830864    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830869    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830872    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830876    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830883    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830886    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830889    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830892    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830895    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830898    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830900    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830903    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830906    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830908    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830911    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830914    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830916    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830919    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830921    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830924    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:32:44.830926 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830927    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830929    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830932    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830935    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830938    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830941    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830944    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830946    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830949    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830951    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830954    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830956    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830959    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830962    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830964    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830966    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830969    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830972    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830974    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830977    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:32:44.831519 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830979    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830982    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830985    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830987    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830990    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830992    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830996    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.830999    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831002    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831005    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831009    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831014    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831017    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831020    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831023    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831026    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831030    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831033    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831037    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:32:44.832015 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831040    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831042    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831045    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831047    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831050    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831053    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831055    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831058    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831061    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831063    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831066    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831073    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831075    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831078    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831080    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831083    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831085    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831088    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831090    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:32:44.832482 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831093    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831095    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831098    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831100    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831104    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831107    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831109    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831112    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831115    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831118    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831120    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831123    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831126    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831128    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831131    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831133    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831136    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831138    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831141    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831143    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:32:44.832966 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831915    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831926    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831930    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831933    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831936    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831939    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831942    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831944    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831947    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831950    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831952    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831955    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831958    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831961    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831963    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831966    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831968    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831971    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831973    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831976    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:32:44.833449 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831978    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831981    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831983    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831986    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831989    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831991    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831994    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831997    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.831999    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832002    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832005    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832007    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832010    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832014    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832017    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832020    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832023    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832026    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832029    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832032    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:32:44.833992 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832034    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832037    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832040    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832044    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832046    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832049    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832052    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832055    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832057    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832061    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832065    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832069    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832071    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832074    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832076    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832079    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832084    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832089    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832092    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:32:44.834479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832095    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832097    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832100    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832102    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832105    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832107    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832111    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832115    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832119    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832122    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832125    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832127    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832130    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832132    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832135    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832137    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832140    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832142    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832145    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:32:44.834946 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832147    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832150    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832152    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832155    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832157    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832160    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832162    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832165    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832167    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832170    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832172    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832175    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832177    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832180    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832182    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832185    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832187    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832190    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832193    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832196    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:32:44.835404 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832199    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832562    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832567    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832569    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832573    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832576    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832578    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832581    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832584    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832586    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832589    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832592    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832594    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832597    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832617    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832620    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832622    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832625    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832628    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:32:44.835898 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832630    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832633    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832636    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832640    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832647    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832649    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832652    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832655    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832657    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832660    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832663    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832665    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832669    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832671    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832674    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832677    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832680    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832682    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832685    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832687    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:32:44.836360 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832690    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832692    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832695    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832697    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832700    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832702    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832705    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832708    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832710    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832713    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832715    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832718    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832720    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832723    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832725    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832728    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832732    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832736    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832739    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:32:44.836873 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832742    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832745    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832748    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832751    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832754    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832757    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832759    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832762    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832765    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832767    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832770    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832772    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832775    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832777    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832780    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832782    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832785    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832787    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832798    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832800    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:32:44.837331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832803    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832807    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832809    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832812    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832815    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832817    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832820    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832822    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832825    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832829    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832831    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832834    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832837    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832839    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832842    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832844    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832847    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832851    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832855    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832858    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:32:44.837831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832860    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.832864    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.832939    2555 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.832974    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.832982    2555 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.832986    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.832989    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.832993    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.832997    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833001    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833004    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833007    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833011    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833014    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833018    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833020    2555 flags.go:64] FLAG: --cgroup-root=""
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833023    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833026    2555 flags.go:64] FLAG: --client-ca-file=""
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833029    2555 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833032    2555 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833036    2555 flags.go:64] FLAG: --cluster-domain=""
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833041    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833044    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833047    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 00:32:44.838306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833050    2555 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833054    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833057    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833059    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833063    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833067    2555 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833070    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833073    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833076    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833079    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833084    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833087    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833090    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833093    2555 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833096    2555 flags.go:64] FLAG: --enable-server="true"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833098    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833104    2555 flags.go:64] FLAG: --event-burst="100"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833107    2555 flags.go:64] FLAG: --event-qps="50"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833109    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833112    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833115    2555 flags.go:64] FLAG: --eviction-hard=""
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833119    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833122    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833124    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833127    2555 flags.go:64] FLAG: --eviction-soft=""
Sep 08 00:32:44.838923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833130    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833133    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833136    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833139    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833143    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833146    2555 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833149    2555 flags.go:64] FLAG: --feature-gates=""
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833152    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833155    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833158    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833161    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833164    2555 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833169    2555 flags.go:64] FLAG: --help="false"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833172    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-142-98.ec2.internal"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833175    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833178    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833181    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833184    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833188    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833191    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833194    2555 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833197    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833199    2555 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 00:32:44.839522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833202    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833205    2555 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833208    2555 flags.go:64] FLAG: --kube-reserved=""
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833211    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833214    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833217    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833219    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833222    2555 flags.go:64] FLAG: --lock-file=""
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833225    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833228    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833231    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833236    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833239    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833243    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833246    2555 flags.go:64] FLAG: --logging-format="text"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833249    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833252    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833255    2555 flags.go:64] FLAG: --manifest-url=""
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833258    2555 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833262    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833265    2555 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833270    2555 flags.go:64] FLAG: --max-pods="110"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833273    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833277    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833279    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 00:32:44.840112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833282    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833285    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833288    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833291    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833298    2555 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833302    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833304    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833307    2555 flags.go:64] FLAG: --pod-cidr=""
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833310    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833315    2555 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833318    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833321    2555 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833324    2555 flags.go:64] FLAG: --port="10250"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833327    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833330    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0287a551380f30f54"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833333    2555 flags.go:64] FLAG: --qos-reserved=""
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833336    2555 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833339    2555 flags.go:64] FLAG: --register-node="true"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833342    2555 flags.go:64] FLAG: --register-with-taints=""
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833345    2555 flags.go:64] FLAG: --registry-burst="10"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833349    2555 flags.go:64] FLAG: --registry-qps="5"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833352    2555 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833355    2555 flags.go:64] FLAG: --reserved-memory=""
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833358    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833362    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 00:32:44.840736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833364    2555 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833367    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833370    2555 flags.go:64] FLAG: --runonce="false"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833373    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833377    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833380    2555 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833383    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833386    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833388    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833391    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833395    2555 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833398    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833401    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833403    2555 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833406    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833409    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833412    2555 flags.go:64] FLAG: --system-cgroups=""
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833415    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833420    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833423    2555 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833426    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833430    2555 flags.go:64] FLAG: --tls-min-version=""
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833432    2555 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833435    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833438    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 00:32:44.841353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833441    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833443    2555 flags.go:64] FLAG: --v="2"
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833448    2555 flags.go:64] FLAG: --version="false"
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833452    2555 flags.go:64] FLAG: --vmodule=""
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833456    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.833460    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833577    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833581    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833584    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833587    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833590    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833594    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833597    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833615    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833618    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833621    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833624    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833627    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833630    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833634    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833637    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:32:44.841965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833641    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833644    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833647    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833650    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833653    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833656    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833658    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833661    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833663    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833666    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833669    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833672    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833674    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833678    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833680    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833683    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833686    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833689    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833691    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833694    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:32:44.842515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833696    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833699    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833703    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833705    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833708    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833710    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833713    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833717    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833721    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833725    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833728    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833735    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833739    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833741    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833744    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833747    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833749    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833752    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833756    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:32:44.843062 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833759    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833762    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833765    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833767    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833770    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833772    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833779    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833782    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833785    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833789    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833791    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833794    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833796    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833799    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833801    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833805    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833808    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833810    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833813    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:32:44.843547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833816    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833818    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833821    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833823    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833826    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833828    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833831    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833834    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833836    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833838    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833841    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833844    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833846    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833849    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833851    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833854    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833856    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833859    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833862    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833866    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:32:44.844300 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833870    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833873    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833876    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833879    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833882    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.833885    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.834375    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.841238    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.841250    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.842034    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.842045    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842103    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842108    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842112    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842115    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:32:44.844904 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842118    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842120    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842123    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842125    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842129    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842131    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842134    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842138    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842142    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842145    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842147    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842150    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842152    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842155    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842158    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842161    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842164    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842167    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842169    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:32:44.845276 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842172    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842174    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842177    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842179    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842182    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842184    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842187    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842189    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842192    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842194    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842197    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842200    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842204    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842207    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842210    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842213    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842216    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842219    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842222    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842224    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:32:44.845750 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842227    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842230    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842232    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842234    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842237    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842239    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842242    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842244    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842247    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842250    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842253    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842255    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842257    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842260    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842263    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842265    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842268    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842270    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842273    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:32:44.846263 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842275    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842278    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842281    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842283    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842286    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842289    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842291    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842294    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842296    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842298    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842301    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842304    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842306    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842309    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842311    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842314    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842317    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842319    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842321    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842324    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:32:44.846793 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842326    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842329    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842332    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842334    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842337    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842339    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842342    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842344    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842347    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842349    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842353    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842356    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842359    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842361    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842363    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842366    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:32:44.847279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842368    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.842373    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842489    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842493    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842497    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842499    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842502    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842505    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842508    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842510    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842513    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842516    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842519    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842521    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:32:44.847682 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842524    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842527    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842530    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842532    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842535    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842538    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842541    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842543    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842546    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842548    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842551    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842553    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842556    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842558    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842561    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842564    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842566    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842569    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842571    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:32:44.848031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842574    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842577    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842579    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842582    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842584    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842587    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842589    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842592    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842595    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842597    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842616    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842619    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842622    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842625    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842627    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842630    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842632    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842635    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842637    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842640    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:32:44.848515 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842642    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842646    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842649    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842653    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842655    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842658    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842661    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842663    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842666    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842668    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842671    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842674    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842677    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842679    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842682    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842685    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842687    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842690    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842693    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:32:44.849028 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842695    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842698    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842700    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842702    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842705    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842707    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842710    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842713    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842715    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842718    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842720    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842722    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842725    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842727    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842730    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842732    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842734    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842737    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842739    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:32:44.849484 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842742    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842744    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842747    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842749    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842752    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842754    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842757    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842759    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842762    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:44.842764    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.842774    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.842875    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.846002    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.846983    2555 server.go:1019] "Starting client certificate rotation"
Sep 08 00:32:44.849965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.847081    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 00:32:44.850332 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.847111    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 00:32:44.850835 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.850820    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 00:32:44.852044 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.852028    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 00:32:44.862730 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.862703    2555 log.go:25] "Validated CRI v1 runtime API"
Sep 08 00:32:44.867858 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.867844    2555 log.go:25] "Validated CRI v1 image API"
Sep 08 00:32:44.868726 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.868711    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 00:32:44.871202 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.871180    2555 fs.go:135] Filesystem UUIDs: map[3b35e503-940e-4696-b241-a3953f7db234:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 909bdce4-581a-41ed-a6c4-d6e74c19086e:/dev/nvme0n1p3]
Sep 08 00:32:44.871250 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.871203    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 00:32:44.875561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.875451    2555 manager.go:216] Machine: {Timestamp:2026-09-08 00:32:44.87496713 +0000 UTC m=+0.354260838 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101596 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273d3c40e37585574a4c8f3e748972 SystemUUID:ec273d3c-40e3-7585-574a-4c8f3e748972 BootID:f47dd023-a945-4f17-89b7-d34bc69eaf7d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a0:38:f8:88:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a0:38:f8:88:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:5a:04:6f:9b:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 00:32:44.875561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.875555    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 00:32:44.875681 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.875651    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 00:32:44.878220 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.878195    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 00:32:44.878383 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.878223    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 00:32:44.878979 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.878969    2555 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 00:32:44.879010 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.878985    2555 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 00:32:44.879010 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.878993    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 00:32:44.879912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.879901    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 00:32:44.881337 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.881326    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 00:32:44.884998 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.884986    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 00:32:44.885141 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885132    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 00:32:44.885171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885160    2555 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 00:32:44.885200 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885179    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 00:32:44.885200 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885198    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 00:32:44.885274 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885211    2555 kubelet.go:398] "Adding apiserver pod source"
Sep 08 00:32:44.885274 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885223    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 00:32:44.885837 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885821    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 00:32:44.885916 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.885848    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 00:32:44.888427 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.888396    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 00:32:44.891552 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.891532    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 00:32:44.893713 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893700    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 00:32:44.893771 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893730    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 00:32:44.893878 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893869    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 00:32:44.893912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893883    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 00:32:44.893912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893889    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 00:32:44.893912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893895    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 00:32:44.893912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893903    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 00:32:44.893912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.893909    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 00:32:44.894678 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.894667    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 00:32:44.894708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.894678    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 00:32:44.894708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.894686    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 00:32:44.894708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.894692    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 00:32:44.894708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.894700    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 00:32:44.894708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.894709    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 00:32:44.895585 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.895573    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 00:32:44.895627 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.895588    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 00:32:44.897571 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.897551    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 00:32:44.897640 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.897557    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 00:32:44.897640 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.897582    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 00:32:44.899512 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.899498    2555 server.go:1268] "Started kubelet"
Sep 08 00:32:44.899635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.899575    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 00:32:44.899738 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.899650    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 00:32:44.899738 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.899586    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 00:32:44.900404 ip-10-0-142-98 systemd[1]: Started Kubernetes Kubelet.
Sep 08 00:32:44.900967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.900949    2555 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 00:32:44.901046 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.901037    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 00:32:44.904370 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.904354    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 00:32:44.905840 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.905772    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 00:32:44.906216 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.906200    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 00:32:44.906804 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.906295    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:44.906998 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.906981    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 00:32:44.907078 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907002    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 00:32:44.907078 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907021    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 00:32:44.907176 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907128    2555 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 00:32:44.907176 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907137    2555 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 00:32:44.907741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907716    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 00:32:44.907741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907730    2555 factory.go:55] Registering systemd factory
Sep 08 00:32:44.907741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907736    2555 factory.go:223] Registration of the systemd container factory successfully
Sep 08 00:32:44.907953 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907940    2555 factory.go:153] Registering CRI-O factory
Sep 08 00:32:44.908013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907958    2555 factory.go:223] Registration of the crio container factory successfully
Sep 08 00:32:44.908013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907974    2555 factory.go:103] Registering Raw factory
Sep 08 00:32:44.908013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.907987    2555 manager.go:1196] Started watching for new ooms in manager
Sep 08 00:32:44.908343 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.908331    2555 manager.go:349] Starting recovery of all containers
Sep 08 00:32:44.915973 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.915947    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 00:32:44.915973 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.915955    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 00:32:44.920284 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.916073    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-98.ec2.internal.18d331315242831b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-98.ec2.internal,UID:ip-10-0-142-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-98.ec2.internal,},FirstTimestamp:2026-09-08 00:32:44.899468059 +0000 UTC m=+0.378761766,LastTimestamp:2026-09-08 00:32:44.899468059 +0000 UTC m=+0.378761766,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-98.ec2.internal,}"
Sep 08 00:32:44.924251 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.924089    2555 manager.go:354] Recovery completed
Sep 08 00:32:44.929141 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.929126    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:32:44.931154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.931135    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:32:44.931226 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.931161    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:32:44.931226 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.931171    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:32:44.931967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.931950    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 00:32:44.932105 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.931968    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 00:32:44.932105 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.931974    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:32:44.932105 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.931987    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 00:32:44.933322 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.933240    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-98.ec2.internal.18d331315425fae6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-98.ec2.internal,UID:ip-10-0-142-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-98.ec2.internal,},FirstTimestamp:2026-09-08 00:32:44.931152614 +0000 UTC m=+0.410446320,LastTimestamp:2026-09-08 00:32:44.931152614 +0000 UTC m=+0.410446320,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-98.ec2.internal,}"
Sep 08 00:32:44.933778 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.933763    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:32:44.934056 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.933856    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:32:44.934056 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.933878    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:32:44.935285 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.935270    2555 policy_none.go:49] "None policy: Start"
Sep 08 00:32:44.935342 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.935290    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 00:32:44.935342 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.935304    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 00:32:44.937454 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.937441    2555 policy_none.go:47] "Start"
Sep 08 00:32:44.942580 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.942475    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-98.ec2.internal.18d3313154262e5a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-98.ec2.internal,UID:ip-10-0-142-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-98.ec2.internal,},FirstTimestamp:2026-09-08 00:32:44.931165786 +0000 UTC m=+0.410459492,LastTimestamp:2026-09-08 00:32:44.931165786 +0000 UTC m=+0.410459492,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-98.ec2.internal,}"
Sep 08 00:32:44.942945 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.942929    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2kl7m"
Sep 08 00:32:44.945551 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.945468    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-98.ec2.internal.18d3313154265366  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-98.ec2.internal,UID:ip-10-0-142-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-98.ec2.internal,},FirstTimestamp:2026-09-08 00:32:44.93117527 +0000 UTC m=+0.410468977,LastTimestamp:2026-09-08 00:32:44.93117527 +0000 UTC m=+0.410468977,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-98.ec2.internal,}"
Sep 08 00:32:44.951532 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.951510    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2kl7m"
Sep 08 00:32:44.955931 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.955854    2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-98.ec2.internal.18d331315425fae6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-98.ec2.internal.18d331315425fae6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-98.ec2.internal,UID:ip-10-0-142-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-98.ec2.internal,},FirstTimestamp:2026-09-08 00:32:44.931152614 +0000 UTC m=+0.410446320,LastTimestamp:2026-09-08 00:32:44.933840828 +0000 UTC m=+0.413134540,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-98.ec2.internal,}"
Sep 08 00:32:44.969218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.969202    2555 manager.go:337] "Starting Device Plugin manager"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.969235    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.969248    2555 server.go:87] "Starting device plugin registration server"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.969569    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.969579    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.969715    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.969949    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:44.969962    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.972032    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 00:32:44.989637 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:44.972079    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:45.009094 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.009070    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 00:32:45.010273 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.010254    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 00:32:45.010346 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.010302    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 00:32:45.010346 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.010322    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 00:32:45.010434 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.010398    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 00:32:45.013110 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.013092    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:32:45.070241 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.070208    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:32:45.071188 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.071163    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:32:45.071271 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.071196    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:32:45.071271 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.071211    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:32:45.071271 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.071242    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.086726 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.086652    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.086726 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.086684    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-98.ec2.internal\": node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:45.110964 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.110938    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal"]
Sep 08 00:32:45.111060 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.111000    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:32:45.111942 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.111928    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:32:45.112013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.111954    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:32:45.112013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.111964    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:32:45.114311 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.114300    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:32:45.114441 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.114428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.114477 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.114453    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:32:45.115318 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.115303    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:32:45.115398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.115325    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:32:45.115398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.115337    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:32:45.115398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.115340    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:32:45.115398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.115361    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:32:45.115398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.115372    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:32:45.117562 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.117547    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.117674 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.117570    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:32:45.118387 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.118369    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:32:45.118491 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.118396    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:32:45.118491 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.118408    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:32:45.133133 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.133114    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-98.ec2.internal\" not found" node="ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.135760 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.135741    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:45.136487 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.136469    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-98.ec2.internal\" not found" node="ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.208842 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.208815    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/458cfede6a20a769684250272677fe1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal\" (UID: \"458cfede6a20a769684250272677fe1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.208986 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.208845    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/458cfede6a20a769684250272677fe1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal\" (UID: \"458cfede6a20a769684250272677fe1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.208986 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.208868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b002b2444c73b6b5500ec2ab14c9652-config\") pod \"kube-apiserver-proxy-ip-10-0-142-98.ec2.internal\" (UID: \"4b002b2444c73b6b5500ec2ab14c9652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.236198 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.236173    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:45.309324 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.309293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/458cfede6a20a769684250272677fe1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal\" (UID: \"458cfede6a20a769684250272677fe1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.309447 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.309371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/458cfede6a20a769684250272677fe1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal\" (UID: \"458cfede6a20a769684250272677fe1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.309447 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.309382    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/458cfede6a20a769684250272677fe1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal\" (UID: \"458cfede6a20a769684250272677fe1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.309447 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.309422    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/458cfede6a20a769684250272677fe1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal\" (UID: \"458cfede6a20a769684250272677fe1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.309570 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.309418    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b002b2444c73b6b5500ec2ab14c9652-config\") pod \"kube-apiserver-proxy-ip-10-0-142-98.ec2.internal\" (UID: \"4b002b2444c73b6b5500ec2ab14c9652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.309570 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.309479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b002b2444c73b6b5500ec2ab14c9652-config\") pod \"kube-apiserver-proxy-ip-10-0-142-98.ec2.internal\" (UID: \"4b002b2444c73b6b5500ec2ab14c9652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.336455 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.336431    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:45.436314 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.436240    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.437322 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.437304    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:45.439510 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:45.439493    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:45.538304 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:45.538272    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:46.774757 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:46.774726    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:46.875364 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:46.875293    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-98.ec2.internal\" not found"
Sep 08 00:32:46.929556 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.929529    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:32:46.943859 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.943842    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:46.969938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.969913    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 00:32:46.971720 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.971706    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal"
Sep 08 00:32:46.981865 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.981846    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 00:32:46.982872 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.982856    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 00:32:46.982977 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.982958    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 00:32:46.983023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:46.982993    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 00:32:47.022185 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.022156    2555 apiserver.go:52] "Watching apiserver"
Sep 08 00:32:47.042323 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.042300    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 00:32:47.042934 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.042917    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 00:32:47.046703 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.046671    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-txvvf","openshift-network-operator/iptables-alerter-5kdkg","kube-system/global-pull-secret-syncer-fc6wt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c","openshift-image-registry/node-ca-pgjp4","openshift-multus/network-metrics-daemon-gmnq8","openshift-network-diagnostics/network-check-target-hv2nm","openshift-ovn-kubernetes/ovnkube-node-rx52p","kube-system/konnectivity-agent-ph5tr","kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qtqzt","openshift-dns/node-resolver-bcfgd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal","openshift-multus/multus-additional-cni-plugins-m6nxh"]
Sep 08 00:32:47.049199 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.049178    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.051325 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.051306    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.053442 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.053424    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.053517 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.053494    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:32:47.055367 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.055350    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 00:32:47.055632 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.055611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.055683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.055657    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qctq4\""
Sep 08 00:32:47.056285 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.056269    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 00:32:47.056366 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.056352    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 00:32:47.056429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.056411    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 00:32:47.057802 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.057780    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.059991 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.059969    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:47.060077 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.060035    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:32:47.060315 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.060299    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 00:32:47.062125 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.062108    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:47.062210 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.062171    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:32:47.063084 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.063066    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 00:32:47.064372 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.064356    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.064756 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.064738    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 00:32:47.064853 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.064760    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 00:32:47.064908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.064892    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n9h4v\""
Sep 08 00:32:47.065027 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.065011    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 00:32:47.065087 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.065075    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rf2ln\""
Sep 08 00:32:47.065128 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.065011    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 00:32:47.065197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.065181    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 00:32:47.065307 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.065294    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 00:32:47.065403 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.065388    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dnbtz\""
Sep 08 00:32:47.067000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.066982    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.069753 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.069733    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.072234 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.072212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.074435 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.074416    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.075568 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.075550    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 00:32:47.076686 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.076663    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 00:32:47.077122 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.077108    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 00:32:47.080340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.080188    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 00:32:47.080340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.080189    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 00:32:47.080340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.080227    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 00:32:47.080340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.080241    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 00:32:47.081446 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.081426    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 00:32:47.081715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.081696    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 00:32:47.081715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.081714    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 00:32:47.081876 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.081724    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 00:32:47.081876 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.081706    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxjq8\""
Sep 08 00:32:47.081876 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.081754    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-45sc2\""
Sep 08 00:32:47.083201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.082998    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xzxc5\""
Sep 08 00:32:47.083201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.083012    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mln59\""
Sep 08 00:32:47.083201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.083060    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 00:32:47.083201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.083082    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 00:32:47.083201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.083005    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hhc5p\""
Sep 08 00:32:47.083201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.083101    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 00:32:47.083807 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.083791    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 00:32:47.090398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.090372    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 00:27:46 +0000 UTC" deadline="2028-04-19 13:47:33.898060281 +0000 UTC"
Sep 08 00:32:47.090398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.090395    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14149h14m46.807667122s"
Sep 08 00:32:47.094113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.094098    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 00:32:47.111190 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.111169    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jh5t2"
Sep 08 00:32:47.121804 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.121787    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jh5t2"
Sep 08 00:32:47.144522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.144506    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 00:32:47.148564 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.148664 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-run\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.148664 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148606    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-k8s-cni-cncf-io\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.148664 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148632    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-daemon-config\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.148664 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148656    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-etc-selinux\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148686    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-lib-modules\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysctl-d\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-system-cni-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9425672-6ac0-4db4-b22e-72a60cacd375-cni-binary-copy\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148816    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-sys-fs\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148832    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d01212a-e314-4028-a478-4fd6332e552d-host\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.148863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148853    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-run-netns\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148890    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-etc-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148918    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-system-cni-dir\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148938    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-kubelet\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148966    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-systemd\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148978    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-netns\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.148991    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-hostroot\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149003    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-dbus\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4qk\" (UniqueName: \"kubernetes.io/projected/bd337a55-7af0-4b6c-98e9-a4980713cb79-kube-api-access-ng4qk\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149039    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5fea2dc6-69d9-4a32-8d6e-de47f938f884-agent-certs\") pod \"konnectivity-agent-ph5tr\" (UID: \"5fea2dc6-69d9-4a32-8d6e-de47f938f884\") " pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149052    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-systemd-units\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149090    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bg79\" (UniqueName: \"kubernetes.io/projected/1470ceeb-9791-406f-a1c6-f00178bf6246-kube-api-access-9bg79\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.149109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjt4\" (UniqueName: \"kubernetes.io/projected/ec584546-9bce-4d84-916d-92b08204d7ce-kube-api-access-ncjt4\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-cnibin\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149138    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-multus-certs\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149153    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-etc-kubernetes\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149188    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-socket-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149200    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-systemd\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149216    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-cni-bin\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149245    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovn-node-metrics-cert\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149271    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-var-lib-kubelet\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149301    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-modprobe-d\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149320    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysconfig\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-kubernetes\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149357    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysctl-conf\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149376    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-conf-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-tmp-dir\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149423    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovnkube-config\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.149489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149443    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldht9\" (UniqueName: \"kubernetes.io/projected/724f7c23-3ce3-45dd-932c-c8e0ce407e83-kube-api-access-ldht9\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149457    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec584546-9bce-4d84-916d-92b08204d7ce-tmp\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149471    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-cni-bin\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fss8m\" (UniqueName: \"kubernetes.io/projected/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-kube-api-access-fss8m\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149505    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-iptables-alerter-script\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-host-slash\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149529    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxs27\" (UniqueName: \"kubernetes.io/projected/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-kube-api-access-jxs27\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149556    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-node-log\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149587    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-hosts-file\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149648    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-cni-netd\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-host\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149710    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5fea2dc6-69d9-4a32-8d6e-de47f938f884-konnectivity-ca\") pod \"konnectivity-agent-ph5tr\" (UID: \"5fea2dc6-69d9-4a32-8d6e-de47f938f884\") " pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149727    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-cnibin\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.150076 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-device-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149806    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tq7g\" (UniqueName: \"kubernetes.io/projected/9d01212a-e314-4028-a478-4fd6332e552d-kube-api-access-5tq7g\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-slash\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-env-overrides\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149879    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-log-socket\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec584546-9bce-4d84-916d-92b08204d7ce-etc-tuned\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149923    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-kubelet-config\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149936    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-registration-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149950    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.149985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ppg\" (UniqueName: \"kubernetes.io/projected/c9425672-6ac0-4db4-b22e-72a60cacd375-kube-api-access-57ppg\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150004    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-var-lib-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-ovn\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150035    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovnkube-script-lib\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-sys\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150079    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-cni-multus\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150093    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.150504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.151013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-cni-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.151013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-socket-dir-parent\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.151013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150185    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-kubelet\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.151013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv99b\" (UniqueName: \"kubernetes.io/projected/dc409628-ba63-48b6-ba42-5c86af963c50-kube-api-access-kv99b\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.151013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.150229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d01212a-e314-4028-a478-4fd6332e552d-serviceca\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.172272 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.172240    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b002b2444c73b6b5500ec2ab14c9652.slice/crio-61457589db12a47d2e583085ef8897234f54b712110f27dc54ad84ba19212fe3 WatchSource:0}: Error finding container 61457589db12a47d2e583085ef8897234f54b712110f27dc54ad84ba19212fe3: Status 404 returned error can't find the container with id 61457589db12a47d2e583085ef8897234f54b712110f27dc54ad84ba19212fe3
Sep 08 00:32:47.172514 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.172491    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458cfede6a20a769684250272677fe1d.slice/crio-cdd68958ec460bbd0fae547db35ef9983d79b83c25881aedc634623a6b2f17bb WatchSource:0}: Error finding container cdd68958ec460bbd0fae547db35ef9983d79b83c25881aedc634623a6b2f17bb: Status 404 returned error can't find the container with id cdd68958ec460bbd0fae547db35ef9983d79b83c25881aedc634623a6b2f17bb
Sep 08 00:32:47.178636 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.178619    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:32:47.230920 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.230893    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:32:47.250908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.250883    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-slash\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.250989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.250910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-env-overrides\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.250989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.250928    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-log-socket\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.250989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.250942    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec584546-9bce-4d84-916d-92b08204d7ce-etc-tuned\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.250989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.250956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-kubelet-config\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.250989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.250971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-registration-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.250989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.250987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ppg\" (UniqueName: \"kubernetes.io/projected/c9425672-6ac0-4db4-b22e-72a60cacd375-kube-api-access-57ppg\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251006    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-log-socket\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251024    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-slash\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-var-lib-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251072    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-registration-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251077    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-kubelet-config\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-ovn\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-ovn\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251132    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovnkube-script-lib\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-sys\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-cni-multus\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251231    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-var-lib-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251238    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-cni-multus\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.251279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-cni-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-socket-dir-parent\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251318    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-sys\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-kubelet\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv99b\" (UniqueName: \"kubernetes.io/projected/dc409628-ba63-48b6-ba42-5c86af963c50-kube-api-access-kv99b\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-kubelet\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-socket-dir-parent\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d01212a-e314-4028-a478-4fd6332e552d-serviceca\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-cni-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-run\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251510    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-k8s-cni-cncf-io\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-daemon-config\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251519    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251550    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-run\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251650    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-k8s-cni-cncf-io\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.251656    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:47.252000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-etc-selinux\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251712    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-lib-modules\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.251729    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:47.751708098 +0000 UTC m=+2.095011113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysctl-d\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-system-cni-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251806    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9425672-6ac0-4db4-b22e-72a60cacd375-cni-binary-copy\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251830    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-sys-fs\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-lib-modules\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d01212a-e314-4028-a478-4fd6332e552d-host\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251879    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d01212a-e314-4028-a478-4fd6332e552d-serviceca\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-run-netns\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-etc-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-etc-selinux\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251938    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-run-netns\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.251978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-etc-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.252845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252003    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-system-cni-dir\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252046    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-kubelet\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252062    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysctl-d\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-systemd\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-netns\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-hostroot\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252163    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-kubelet\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252173    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-daemon-config\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252215    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-sys-fs\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-netns\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252175    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-dbus\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252238    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-dbus\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d01212a-e314-4028-a478-4fd6332e552d-host\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252264    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4qk\" (UniqueName: \"kubernetes.io/projected/bd337a55-7af0-4b6c-98e9-a4980713cb79-kube-api-access-ng4qk\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovnkube-script-lib\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252292    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5fea2dc6-69d9-4a32-8d6e-de47f938f884-agent-certs\") pod \"konnectivity-agent-ph5tr\" (UID: \"5fea2dc6-69d9-4a32-8d6e-de47f938f884\") " pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252294    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-system-cni-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.253634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252319    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-systemd-units\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252294    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-system-cni-dir\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252346    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252366    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-systemd-units\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bg79\" (UniqueName: \"kubernetes.io/projected/1470ceeb-9791-406f-a1c6-f00178bf6246-kube-api-access-9bg79\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-hostroot\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjt4\" (UniqueName: \"kubernetes.io/projected/ec584546-9bce-4d84-916d-92b08204d7ce-kube-api-access-ncjt4\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9425672-6ac0-4db4-b22e-72a60cacd375-cni-binary-copy\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-systemd\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252493    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252531    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-cnibin\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-multus-certs\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-etc-kubernetes\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252623    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-cnibin\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252629    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-socket-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252675    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-run-multus-certs\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252679    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-etc-kubernetes\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.254429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-systemd\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252720    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-systemd\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-cni-bin\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252750    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovn-node-metrics-cert\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252750    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-cni-bin\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-var-lib-kubelet\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252793    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-socket-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252819    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-env-overrides\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252831    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-run-openvswitch\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252831    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-modprobe-d\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252850    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc409628-ba63-48b6-ba42-5c86af963c50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-var-lib-kubelet\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252876    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysconfig\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-modprobe-d\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-kubernetes\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysconfig\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.254997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252939    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysctl-conf\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-kubernetes\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-conf-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.252984    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-tmp-dir\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovnkube-config\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253008    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-multus-conf-dir\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253035    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldht9\" (UniqueName: \"kubernetes.io/projected/724f7c23-3ce3-45dd-932c-c8e0ce407e83-kube-api-access-ldht9\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-etc-sysctl-conf\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253058    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec584546-9bce-4d84-916d-92b08204d7ce-tmp\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253091    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-cni-bin\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fss8m\" (UniqueName: \"kubernetes.io/projected/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-kube-api-access-fss8m\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-iptables-alerter-script\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253171    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-host-slash\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253221    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxs27\" (UniqueName: \"kubernetes.io/projected/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-kube-api-access-jxs27\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253247    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-node-log\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-hosts-file\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-cni-netd\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.255635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253345    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-host\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5fea2dc6-69d9-4a32-8d6e-de47f938f884-konnectivity-ca\") pod \"konnectivity-agent-ph5tr\" (UID: \"5fea2dc6-69d9-4a32-8d6e-de47f938f884\") " pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-cnibin\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9425672-6ac0-4db4-b22e-72a60cacd375-host-var-lib-cni-bin\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253473    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-device-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tq7g\" (UniqueName: \"kubernetes.io/projected/9d01212a-e314-4028-a478-4fd6332e552d-kube-api-access-5tq7g\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253371    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-host-slash\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-cnibin\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc409628-ba63-48b6-ba42-5c86af963c50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254039    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-iptables-alerter-script\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254087    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1470ceeb-9791-406f-a1c6-f00178bf6246-device-dir\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.253472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovnkube-config\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254120    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5fea2dc6-69d9-4a32-8d6e-de47f938f884-konnectivity-ca\") pod \"konnectivity-agent-ph5tr\" (UID: \"5fea2dc6-69d9-4a32-8d6e-de47f938f884\") " pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-host-cni-netd\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.256154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254183    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec584546-9bce-4d84-916d-92b08204d7ce-host\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254192    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-hosts-file\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254225    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/724f7c23-3ce3-45dd-932c-c8e0ce407e83-node-log\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.254259    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.254330    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret podName:7e5f18e2-434c-4068-9d88-d5e4f5f10a8f nodeName:}" failed. No retries permitted until 2026-09-08 00:32:47.754312392 +0000 UTC m=+2.097615396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret") pod "global-pull-secret-syncer-fc6wt" (UID: "7e5f18e2-434c-4068-9d88-d5e4f5f10a8f") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.254413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-tmp-dir\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.255416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec584546-9bce-4d84-916d-92b08204d7ce-tmp\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.255452    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec584546-9bce-4d84-916d-92b08204d7ce-etc-tuned\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.255459    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/724f7c23-3ce3-45dd-932c-c8e0ce407e83-ovn-node-metrics-cert\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.256639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.255695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5fea2dc6-69d9-4a32-8d6e-de47f938f884-agent-certs\") pod \"konnectivity-agent-ph5tr\" (UID: \"5fea2dc6-69d9-4a32-8d6e-de47f938f884\") " pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.262322 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.262306    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:32:47.278011 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.277989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv99b\" (UniqueName: \"kubernetes.io/projected/dc409628-ba63-48b6-ba42-5c86af963c50-kube-api-access-kv99b\") pod \"multus-additional-cni-plugins-m6nxh\" (UID: \"dc409628-ba63-48b6-ba42-5c86af963c50\") " pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.279926 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.279906    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fss8m\" (UniqueName: \"kubernetes.io/projected/3460b5c1-7a31-4b5a-83df-5ff3ea4eac70-kube-api-access-fss8m\") pod \"node-resolver-bcfgd\" (UID: \"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70\") " pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.280022 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.279907    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:32:47.280022 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.279960    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:32:47.280022 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.279975    2555 projected.go:196] Error preparing data for projected volume kube-api-access-5jdxn for pod openshift-network-diagnostics/network-check-target-hv2nm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:47.280144 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.280027    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn podName:e4e65bf2-ac1c-4438-a55f-e9984cae1b61 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:47.780010628 +0000 UTC m=+2.123313632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5jdxn" (UniqueName: "kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn") pod "network-check-target-hv2nm" (UID: "e4e65bf2-ac1c-4438-a55f-e9984cae1b61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:47.280411 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.280391    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4qk\" (UniqueName: \"kubernetes.io/projected/bd337a55-7af0-4b6c-98e9-a4980713cb79-kube-api-access-ng4qk\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:47.281938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.281915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bg79\" (UniqueName: \"kubernetes.io/projected/1470ceeb-9791-406f-a1c6-f00178bf6246-kube-api-access-9bg79\") pod \"aws-ebs-csi-driver-node-4tl9c\" (UID: \"1470ceeb-9791-406f-a1c6-f00178bf6246\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.283369 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.283352    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ppg\" (UniqueName: \"kubernetes.io/projected/c9425672-6ac0-4db4-b22e-72a60cacd375-kube-api-access-57ppg\") pod \"multus-txvvf\" (UID: \"c9425672-6ac0-4db4-b22e-72a60cacd375\") " pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.285156 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.285129    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjt4\" (UniqueName: \"kubernetes.io/projected/ec584546-9bce-4d84-916d-92b08204d7ce-kube-api-access-ncjt4\") pod \"tuned-qtqzt\" (UID: \"ec584546-9bce-4d84-916d-92b08204d7ce\") " pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.285245 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.285229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxs27\" (UniqueName: \"kubernetes.io/projected/42522b92-a3ef-4c7c-a8ca-0a9a8a16756d-kube-api-access-jxs27\") pod \"iptables-alerter-5kdkg\" (UID: \"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d\") " pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.288786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.288760    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tq7g\" (UniqueName: \"kubernetes.io/projected/9d01212a-e314-4028-a478-4fd6332e552d-kube-api-access-5tq7g\") pod \"node-ca-pgjp4\" (UID: \"9d01212a-e314-4028-a478-4fd6332e552d\") " pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.288845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.288763    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldht9\" (UniqueName: \"kubernetes.io/projected/724f7c23-3ce3-45dd-932c-c8e0ce407e83-kube-api-access-ldht9\") pod \"ovnkube-node-rx52p\" (UID: \"724f7c23-3ce3-45dd-932c-c8e0ce407e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.379675 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.379578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-txvvf"
Sep 08 00:32:47.386905 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.386888    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5kdkg"
Sep 08 00:32:47.389273 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.389250    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9425672_6ac0_4db4_b22e_72a60cacd375.slice/crio-676386cd0c02e02573e84ed651366f91973bf0b5cfb8b89c4c1f429b6a409a8b WatchSource:0}: Error finding container 676386cd0c02e02573e84ed651366f91973bf0b5cfb8b89c4c1f429b6a409a8b: Status 404 returned error can't find the container with id 676386cd0c02e02573e84ed651366f91973bf0b5cfb8b89c4c1f429b6a409a8b
Sep 08 00:32:47.395167 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.395145    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42522b92_a3ef_4c7c_a8ca_0a9a8a16756d.slice/crio-8cc1909b58090c6a5b163507813350b240387e291f0321f4a4f059a508b3f0f6 WatchSource:0}: Error finding container 8cc1909b58090c6a5b163507813350b240387e291f0321f4a4f059a508b3f0f6: Status 404 returned error can't find the container with id 8cc1909b58090c6a5b163507813350b240387e291f0321f4a4f059a508b3f0f6
Sep 08 00:32:47.399860 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.399845    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c"
Sep 08 00:32:47.408837 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.408816    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1470ceeb_9791_406f_a1c6_f00178bf6246.slice/crio-3adee96dacabee72ec3bca8587e0fa70ca37183e7a435ad8f054f6a15bcec054 WatchSource:0}: Error finding container 3adee96dacabee72ec3bca8587e0fa70ca37183e7a435ad8f054f6a15bcec054: Status 404 returned error can't find the container with id 3adee96dacabee72ec3bca8587e0fa70ca37183e7a435ad8f054f6a15bcec054
Sep 08 00:32:47.410425 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.410409    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgjp4"
Sep 08 00:32:47.418303 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.418285    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d01212a_e314_4028_a478_4fd6332e552d.slice/crio-a3b23c815f9d62c867771a604247d8c70aafd04f415de18e648e293764fcdf90 WatchSource:0}: Error finding container a3b23c815f9d62c867771a604247d8c70aafd04f415de18e648e293764fcdf90: Status 404 returned error can't find the container with id a3b23c815f9d62c867771a604247d8c70aafd04f415de18e648e293764fcdf90
Sep 08 00:32:47.424432 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.424416    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:32:47.431934 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.431910    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724f7c23_3ce3_45dd_932c_c8e0ce407e83.slice/crio-c469484811711b1b3ab8de438d40739dad97f95540528cf08f72ef53662eb651 WatchSource:0}: Error finding container c469484811711b1b3ab8de438d40739dad97f95540528cf08f72ef53662eb651: Status 404 returned error can't find the container with id c469484811711b1b3ab8de438d40739dad97f95540528cf08f72ef53662eb651
Sep 08 00:32:47.439011 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.438997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:32:47.444224 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.444205    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qtqzt"
Sep 08 00:32:47.446582 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.446560    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fea2dc6_69d9_4a32_8d6e_de47f938f884.slice/crio-73d3626d08ec5bc10ab0bfcee3f399b580c4ea7c78723459792e4551440d366e WatchSource:0}: Error finding container 73d3626d08ec5bc10ab0bfcee3f399b580c4ea7c78723459792e4551440d366e: Status 404 returned error can't find the container with id 73d3626d08ec5bc10ab0bfcee3f399b580c4ea7c78723459792e4551440d366e
Sep 08 00:32:47.452429 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.452405    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec584546_9bce_4d84_916d_92b08204d7ce.slice/crio-747908965c4ce408a29020c09fa858b68e9825c463607f7492b9e042844a025d WatchSource:0}: Error finding container 747908965c4ce408a29020c09fa858b68e9825c463607f7492b9e042844a025d: Status 404 returned error can't find the container with id 747908965c4ce408a29020c09fa858b68e9825c463607f7492b9e042844a025d
Sep 08 00:32:47.454941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.454902    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bcfgd"
Sep 08 00:32:47.459684 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.459660    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m6nxh"
Sep 08 00:32:47.464119 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.463968    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3460b5c1_7a31_4b5a_83df_5ff3ea4eac70.slice/crio-50ce98d986c932dc86d6c0dcbd1c2bd2ec3d4acda44e85f596d9cf6ca7e4f1b5 WatchSource:0}: Error finding container 50ce98d986c932dc86d6c0dcbd1c2bd2ec3d4acda44e85f596d9cf6ca7e4f1b5: Status 404 returned error can't find the container with id 50ce98d986c932dc86d6c0dcbd1c2bd2ec3d4acda44e85f596d9cf6ca7e4f1b5
Sep 08 00:32:47.468686 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:32:47.468658    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc409628_ba63_48b6_ba42_5c86af963c50.slice/crio-71da2803ac2c8f6e41a0d459c6650a2176b0ad4b28af6b47d4e4c869cd8512bc WatchSource:0}: Error finding container 71da2803ac2c8f6e41a0d459c6650a2176b0ad4b28af6b47d4e4c869cd8512bc: Status 404 returned error can't find the container with id 71da2803ac2c8f6e41a0d459c6650a2176b0ad4b28af6b47d4e4c869cd8512bc
Sep 08 00:32:47.758023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.757937    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:47.758023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.758014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:47.758237 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.758200    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:47.758291 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.758266    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:48.758246896 +0000 UTC m=+3.101549900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:47.758674 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.758653    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:47.758770 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.758717    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret podName:7e5f18e2-434c-4068-9d88-d5e4f5f10a8f nodeName:}" failed. No retries permitted until 2026-09-08 00:32:48.758699388 +0000 UTC m=+3.102002409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret") pod "global-pull-secret-syncer-fc6wt" (UID: "7e5f18e2-434c-4068-9d88-d5e4f5f10a8f") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:47.859005 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:47.858950    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:47.859183 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.859134    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:32:47.859183 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.859158    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:32:47.859183 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.859173    2555 projected.go:196] Error preparing data for projected volume kube-api-access-5jdxn for pod openshift-network-diagnostics/network-check-target-hv2nm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:47.859334 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:47.859234    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn podName:e4e65bf2-ac1c-4438-a55f-e9984cae1b61 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:48.859215269 +0000 UTC m=+3.202518289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jdxn" (UniqueName: "kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn") pod "network-check-target-hv2nm" (UID: "e4e65bf2-ac1c-4438-a55f-e9984cae1b61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:48.037731 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.037656    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:32:48.125136 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.125097    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 00:27:47 +0000 UTC" deadline="2028-04-13 05:45:27.17920397 +0000 UTC"
Sep 08 00:32:48.125136 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.125134    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13997h12m39.054074976s"
Sep 08 00:32:48.150087 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.149722    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:48.150087 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.149855    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:32:48.188704 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.188395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerStarted","Data":"71da2803ac2c8f6e41a0d459c6650a2176b0ad4b28af6b47d4e4c869cd8512bc"}
Sep 08 00:32:48.209922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.206799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"c469484811711b1b3ab8de438d40739dad97f95540528cf08f72ef53662eb651"}
Sep 08 00:32:48.217003 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.216969    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5kdkg" event={"ID":"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d","Type":"ContainerStarted","Data":"8cc1909b58090c6a5b163507813350b240387e291f0321f4a4f059a508b3f0f6"}
Sep 08 00:32:48.220110 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.220073    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal" event={"ID":"4b002b2444c73b6b5500ec2ab14c9652","Type":"ContainerStarted","Data":"61457589db12a47d2e583085ef8897234f54b712110f27dc54ad84ba19212fe3"}
Sep 08 00:32:48.227353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.227326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal" event={"ID":"458cfede6a20a769684250272677fe1d","Type":"ContainerStarted","Data":"cdd68958ec460bbd0fae547db35ef9983d79b83c25881aedc634623a6b2f17bb"}
Sep 08 00:32:48.239416 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.239386    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bcfgd" event={"ID":"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70","Type":"ContainerStarted","Data":"50ce98d986c932dc86d6c0dcbd1c2bd2ec3d4acda44e85f596d9cf6ca7e4f1b5"}
Sep 08 00:32:48.243962 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.243320    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qtqzt" event={"ID":"ec584546-9bce-4d84-916d-92b08204d7ce","Type":"ContainerStarted","Data":"747908965c4ce408a29020c09fa858b68e9825c463607f7492b9e042844a025d"}
Sep 08 00:32:48.260334 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.260298    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ph5tr" event={"ID":"5fea2dc6-69d9-4a32-8d6e-de47f938f884","Type":"ContainerStarted","Data":"73d3626d08ec5bc10ab0bfcee3f399b580c4ea7c78723459792e4551440d366e"}
Sep 08 00:32:48.281907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.281852    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgjp4" event={"ID":"9d01212a-e314-4028-a478-4fd6332e552d","Type":"ContainerStarted","Data":"a3b23c815f9d62c867771a604247d8c70aafd04f415de18e648e293764fcdf90"}
Sep 08 00:32:48.285551 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.285522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c" event={"ID":"1470ceeb-9791-406f-a1c6-f00178bf6246","Type":"ContainerStarted","Data":"3adee96dacabee72ec3bca8587e0fa70ca37183e7a435ad8f054f6a15bcec054"}
Sep 08 00:32:48.298121 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.298022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txvvf" event={"ID":"c9425672-6ac0-4db4-b22e-72a60cacd375","Type":"ContainerStarted","Data":"676386cd0c02e02573e84ed651366f91973bf0b5cfb8b89c4c1f429b6a409a8b"}
Sep 08 00:32:48.786031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.785987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:48.786215 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.786069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:48.786215 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.786143    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:48.786337 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.786141    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:48.786337 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.786243    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:50.78618656 +0000 UTC m=+5.129489565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:48.786337 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.786268    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret podName:7e5f18e2-434c-4068-9d88-d5e4f5f10a8f nodeName:}" failed. No retries permitted until 2026-09-08 00:32:50.786257418 +0000 UTC m=+5.129560429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret") pod "global-pull-secret-syncer-fc6wt" (UID: "7e5f18e2-434c-4068-9d88-d5e4f5f10a8f") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:48.887671 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:48.887636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:48.887907 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.887842    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:32:48.887907 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.887864    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:32:48.887907 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.887877    2555 projected.go:196] Error preparing data for projected volume kube-api-access-5jdxn for pod openshift-network-diagnostics/network-check-target-hv2nm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:48.888134 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:48.887949    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn podName:e4e65bf2-ac1c-4438-a55f-e9984cae1b61 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:50.887931376 +0000 UTC m=+5.231234378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jdxn" (UniqueName: "kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn") pod "network-check-target-hv2nm" (UID: "e4e65bf2-ac1c-4438-a55f-e9984cae1b61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:49.126316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:49.126276    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 00:27:47 +0000 UTC" deadline="2028-06-11 21:56:07.373218912 +0000 UTC"
Sep 08 00:32:49.126316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:49.126314    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15429h23m18.246909327s"
Sep 08 00:32:49.146708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:49.146651    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:49.146883 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:49.146835    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:32:49.147220 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:49.147201    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:49.147312 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:49.147285    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:32:49.905518 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:49.905484    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:32:50.149075 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:50.149031    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:50.149540 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.149172    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:32:50.804352 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:50.804317    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:50.804531 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:50.804394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:50.804531 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.804514    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:50.804659 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.804573    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:54.804553571 +0000 UTC m=+9.147856584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:50.804855 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.804746    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:50.804855 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.804826    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret podName:7e5f18e2-434c-4068-9d88-d5e4f5f10a8f nodeName:}" failed. No retries permitted until 2026-09-08 00:32:54.804807369 +0000 UTC m=+9.148110374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret") pod "global-pull-secret-syncer-fc6wt" (UID: "7e5f18e2-434c-4068-9d88-d5e4f5f10a8f") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:50.906093 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:50.905626    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:50.906093 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.905765    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:32:50.906093 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.905783    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:32:50.906093 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.905801    2555 projected.go:196] Error preparing data for projected volume kube-api-access-5jdxn for pod openshift-network-diagnostics/network-check-target-hv2nm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:50.906093 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:50.905842    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn podName:e4e65bf2-ac1c-4438-a55f-e9984cae1b61 nodeName:}" failed. No retries permitted until 2026-09-08 00:32:54.90583035 +0000 UTC m=+9.249133348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jdxn" (UniqueName: "kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn") pod "network-check-target-hv2nm" (UID: "e4e65bf2-ac1c-4438-a55f-e9984cae1b61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:51.147030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:51.146991    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:51.147030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:51.147028    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:51.147257 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:51.147132    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:32:51.147349 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:51.147292    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:32:52.147267 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:52.147222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:52.147791 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:52.147366    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:32:53.147445 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:53.147412    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:53.147940 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:53.147413    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:53.147940 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:53.147542    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:32:53.147940 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:53.147652    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:32:54.147586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:54.147549    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:54.148026 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.147759    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:32:54.839240 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:54.839190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:54.839417 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:54.839275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:54.839417 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.839366    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:54.839559 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.839449    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:02.839429547 +0000 UTC m=+17.182732559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:32:54.839559 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.839454    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:54.839559 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.839511    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret podName:7e5f18e2-434c-4068-9d88-d5e4f5f10a8f nodeName:}" failed. No retries permitted until 2026-09-08 00:33:02.839493389 +0000 UTC m=+17.182796410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret") pod "global-pull-secret-syncer-fc6wt" (UID: "7e5f18e2-434c-4068-9d88-d5e4f5f10a8f") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:32:54.940436 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:54.940395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:54.940635 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.940603    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:32:54.940694 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.940644    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:32:54.940694 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.940660    2555 projected.go:196] Error preparing data for projected volume kube-api-access-5jdxn for pod openshift-network-diagnostics/network-check-target-hv2nm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:54.940784 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:54.940719    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn podName:e4e65bf2-ac1c-4438-a55f-e9984cae1b61 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:02.940700628 +0000 UTC m=+17.284003630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jdxn" (UniqueName: "kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn") pod "network-check-target-hv2nm" (UID: "e4e65bf2-ac1c-4438-a55f-e9984cae1b61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:32:55.147056 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:55.147020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:55.147246 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:55.147068    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:55.147246 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:55.147155    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:32:55.147365 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:55.147284    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:32:56.148164 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:56.148133    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:56.148639 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:56.148227    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:32:57.147665 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:57.147439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:57.147665 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:57.147471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:57.147665 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:57.147571    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:32:57.147930 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:57.147718    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:32:58.147425 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:58.147392    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:32:58.148634 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:58.147548    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:32:59.147308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:59.147270    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:32:59.147498 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:59.147398    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:32:59.147498 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:32:59.147471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:32:59.147949 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:32:59.147606    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:00.147621 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:00.147573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:00.148046 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:00.147717    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:01.147378 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:01.147339    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:01.147534 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:01.147339    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:01.147534 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:01.147479    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:01.147660 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:01.147535    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:02.147139 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:02.147105    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:02.147299 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:02.147207    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:02.900214 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:02.900175    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:02.900712 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:02.900251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:02.900712 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:02.900346    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:33:02.900712 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:02.900351    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:33:02.900712 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:02.900417    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:18.900397401 +0000 UTC m=+33.243700415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:33:02.900712 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:02.900434    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret podName:7e5f18e2-434c-4068-9d88-d5e4f5f10a8f nodeName:}" failed. No retries permitted until 2026-09-08 00:33:18.900427288 +0000 UTC m=+33.243730285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret") pod "global-pull-secret-syncer-fc6wt" (UID: "7e5f18e2-434c-4068-9d88-d5e4f5f10a8f") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:33:03.001236 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:03.001203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:03.001410 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:03.001363    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:33:03.001410 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:03.001381    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:33:03.001410 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:03.001389    2555 projected.go:196] Error preparing data for projected volume kube-api-access-5jdxn for pod openshift-network-diagnostics/network-check-target-hv2nm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:33:03.001566 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:03.001441    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn podName:e4e65bf2-ac1c-4438-a55f-e9984cae1b61 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:19.001423754 +0000 UTC m=+33.344726769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jdxn" (UniqueName: "kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn") pod "network-check-target-hv2nm" (UID: "e4e65bf2-ac1c-4438-a55f-e9984cae1b61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:33:03.147500 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:03.147461    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:03.147693 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:03.147464    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:03.147693 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:03.147587    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:03.147804 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:03.147689    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:04.147363 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:04.147334    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:04.147858 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:04.147465    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:05.146515 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:05.146478    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:05.146722 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:05.146619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:05.146722 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:05.146625    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:05.146844 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:05.146787    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:06.147176 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:06.147138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:06.147736 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:06.147238    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:07.147032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.146726    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:07.147170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.146785    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:07.147170 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:07.147099    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:07.147760 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:07.147188    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:07.345461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.345372    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc409628-ba63-48b6-ba42-5c86af963c50" containerID="f6a8953ebc87b17eaebb60e979062e03416ba27c15713e9cd233c537f8433f6b" exitCode=0
Sep 08 00:33:07.345645 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.345459    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerDied","Data":"f6a8953ebc87b17eaebb60e979062e03416ba27c15713e9cd233c537f8433f6b"}
Sep 08 00:33:07.348558 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.348540    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:33:07.348952 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.348931    2555 generic.go:356] "Generic (PLEG): container finished" podID="724f7c23-3ce3-45dd-932c-c8e0ce407e83" containerID="765f3183531ccee8c153021439ba554e4363689a2c19f95f230df3bd4fcae8fa" exitCode=1
Sep 08 00:33:07.349034 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.348985    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"c9c7f9d8730f701b8cd9ef6a8f1b4a6664b125e822fea07e166eba1a41dbcbe7"}
Sep 08 00:33:07.349081 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.349048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"c8d3422210e48880828df973ce4449b42d708a4be3cce1d920f7c44846c85699"}
Sep 08 00:33:07.349115 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.349081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"5968cebc354004b740fd8b959b4590474383732d9e779a416690eee9c70ba73d"}
Sep 08 00:33:07.349115 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.349109    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"437f77ed571dcd95948c8f04dec3f29f5d329f97e1d84988833aa334af987703"}
Sep 08 00:33:07.349185 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.349148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerDied","Data":"765f3183531ccee8c153021439ba554e4363689a2c19f95f230df3bd4fcae8fa"}
Sep 08 00:33:07.349221 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.349188    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"bd4a4301192878d9e831563454aa51e4b9eee37dd15eaf7580eb5b88948f76f7"}
Sep 08 00:33:07.350485 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.350462    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal" event={"ID":"4b002b2444c73b6b5500ec2ab14c9652","Type":"ContainerStarted","Data":"8f32db65952dd89b343bd41e04a211623fb66820e8eea3c3ceebe6f605490317"}
Sep 08 00:33:07.352480 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.352457    2555 generic.go:356] "Generic (PLEG): container finished" podID="458cfede6a20a769684250272677fe1d" containerID="12ca97bf0661b93debfcfbf747ec7e0dd585f0fdf4c216055c06c9e01f15cbec" exitCode=0
Sep 08 00:33:07.352574 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.352523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal" event={"ID":"458cfede6a20a769684250272677fe1d","Type":"ContainerDied","Data":"12ca97bf0661b93debfcfbf747ec7e0dd585f0fdf4c216055c06c9e01f15cbec"}
Sep 08 00:33:07.354037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.354011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bcfgd" event={"ID":"3460b5c1-7a31-4b5a-83df-5ff3ea4eac70","Type":"ContainerStarted","Data":"399398e0568553ef7b0c6befa7ab1f3ee50ac32875e2f1ce498511aae3e832d0"}
Sep 08 00:33:07.355464 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.355445    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qtqzt" event={"ID":"ec584546-9bce-4d84-916d-92b08204d7ce","Type":"ContainerStarted","Data":"9820702b53c13904f7d972b7699454bbbe3c92b1e4785d7eaf09797ae58a6d6a"}
Sep 08 00:33:07.356914 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.356895    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ph5tr" event={"ID":"5fea2dc6-69d9-4a32-8d6e-de47f938f884","Type":"ContainerStarted","Data":"e5c75db9ffef6cfd43ac6227e5452d3307c84155788165a19bf4aad9bea74d62"}
Sep 08 00:33:07.358195 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.358176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgjp4" event={"ID":"9d01212a-e314-4028-a478-4fd6332e552d","Type":"ContainerStarted","Data":"3075a5688154362ab33f8209a8530d7f2ec855c656c61c50fd8ddec5f79a15ed"}
Sep 08 00:33:07.359522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.359480    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c" event={"ID":"1470ceeb-9791-406f-a1c6-f00178bf6246","Type":"ContainerStarted","Data":"db56a0d74b42d0507305531e4af9e5e169a882d68243e36da7b5e60ea702aeb0"}
Sep 08 00:33:07.360995 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.360978    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-txvvf" event={"ID":"c9425672-6ac0-4db4-b22e-72a60cacd375","Type":"ContainerStarted","Data":"916214f132e601e8fbd9227f70c2a6071e6e71cd6e7d778044caaffb80dbd250"}
Sep 08 00:33:07.422006 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.421950    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ph5tr" podStartSLOduration=2.632296555 podStartE2EDuration="21.421933346s" totalImagesPullingTime="18.789636791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:07.401602283 +0000 UTC m=+21.744905297" watchObservedRunningTime="2026-09-08 00:33:07.421933346 +0000 UTC m=+21.765236365"
Sep 08 00:33:07.463746 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.463696    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pgjp4" podStartSLOduration=2.644599049 podStartE2EDuration="21.463679611s" totalImagesPullingTime="18.819080562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:07.440561987 +0000 UTC m=+21.783865007" watchObservedRunningTime="2026-09-08 00:33:07.463679611 +0000 UTC m=+21.806982673"
Sep 08 00:33:07.482316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.482261    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qtqzt" podStartSLOduration=2.66648932 podStartE2EDuration="21.482240978s" totalImagesPullingTime="18.815751658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:07.463368043 +0000 UTC m=+21.806671063" watchObservedRunningTime="2026-09-08 00:33:07.482240978 +0000 UTC m=+21.825543998"
Sep 08 00:33:07.482451 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.482408    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bcfgd" podStartSLOduration=2.709102703 podStartE2EDuration="21.482400585s" totalImagesPullingTime="18.773297882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:07.482133355 +0000 UTC m=+21.825436385" watchObservedRunningTime="2026-09-08 00:33:07.482400585 +0000 UTC m=+21.825703624"
Sep 08 00:33:07.502703 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.502652    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-98.ec2.internal" podStartSLOduration=21.502635663 podStartE2EDuration="21.502635663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:07.502634563 +0000 UTC m=+21.845937583" watchObservedRunningTime="2026-09-08 00:33:07.502635663 +0000 UTC m=+21.845938681"
Sep 08 00:33:07.529525 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:07.529468    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-txvvf" podStartSLOduration=2.507848323 podStartE2EDuration="21.529452354s" totalImagesPullingTime="19.021604031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:07.528964179 +0000 UTC m=+21.872267226" watchObservedRunningTime="2026-09-08 00:33:07.529452354 +0000 UTC m=+21.872755372"
Sep 08 00:33:08.147205 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.147171    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:08.147973 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:08.147318    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:08.317498 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.317467    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 00:33:08.365217 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.365176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5kdkg" event={"ID":"42522b92-a3ef-4c7c-a8ca-0a9a8a16756d","Type":"ContainerStarted","Data":"b82ef264112b30a061a18f12880878ff8c1b50efaf0f8d2aca4223122b13c1a2"}
Sep 08 00:33:08.367444 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.367413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal" event={"ID":"458cfede6a20a769684250272677fe1d","Type":"ContainerStarted","Data":"943a4a85175700d00051af4e9421ceb281f5cd38e99d5853125d6b4718207480"}
Sep 08 00:33:08.369758 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.369728    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c" event={"ID":"1470ceeb-9791-406f-a1c6-f00178bf6246","Type":"ContainerStarted","Data":"43f05a33403f998a16483aafa3b83bc3f1dc43f846727ed9c79e608f74a9ef28"}
Sep 08 00:33:08.388522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.388475    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5kdkg" podStartSLOduration=3.546068989 podStartE2EDuration="22.388454772s" totalImagesPullingTime="18.842385783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:08.388200088 +0000 UTC m=+22.731503107" watchObservedRunningTime="2026-09-08 00:33:08.388454772 +0000 UTC m=+22.731757792"
Sep 08 00:33:08.568112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.567932    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:33:08.568716 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.568696    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:33:08.585835 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:08.585786    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-98.ec2.internal" podStartSLOduration=22.585770692 podStartE2EDuration="22.585770692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:08.403826372 +0000 UTC m=+22.747129392" watchObservedRunningTime="2026-09-08 00:33:08.585770692 +0000 UTC m=+22.929073711"
Sep 08 00:33:09.118762 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.118583    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T00:33:08.31749246Z","UUID":"7c790383-5cce-4026-9b3f-66e1cdc62082","Handler":null,"Name":"","Endpoint":""}
Sep 08 00:33:09.120901 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.120873    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 00:33:09.120901 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.120907    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 00:33:09.146700 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.146547    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:09.146700 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:09.146697    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:09.146925 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.146780    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:09.146925 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:09.146862    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:09.377072 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.377035    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:33:09.377669 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.377630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"8920dff53547e3bcc65391b9498a9fe071baabae3d506f46679e5016064a0384"}
Sep 08 00:33:09.378029 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.377998    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:33:09.378425 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:09.378407    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ph5tr"
Sep 08 00:33:10.147170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:10.147138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:10.147337 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:10.147309    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:10.382474 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:10.382394    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c" event={"ID":"1470ceeb-9791-406f-a1c6-f00178bf6246","Type":"ContainerStarted","Data":"550ea1687a4c0455e828526f9d1dde2f70a16d1485b7fca9eb931fad35e36281"}
Sep 08 00:33:11.147419 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:11.147389    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:11.147419 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:11.147404    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:11.147674 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:11.147510    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:11.147674 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:11.147653    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:12.147557 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:12.147525    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:12.147998 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:12.147692    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:12.390522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:12.390500    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:33:12.390957 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:12.390933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"95cb5ece7528e45be1b2ea520844eeb3f757183d106d62d5788c3497132be61b"}
Sep 08 00:33:12.394770 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:12.394631    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:33:12.394846 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:12.394788    2555 scope.go:117] "RemoveContainer" containerID="765f3183531ccee8c153021439ba554e4363689a2c19f95f230df3bd4fcae8fa"
Sep 08 00:33:12.414997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:12.414900    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:33:12.434117 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:12.433896    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4tl9c" podStartSLOduration=4.609776493 podStartE2EDuration="26.43387654s" totalImagesPullingTime="21.824100047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:10.432519283 +0000 UTC m=+24.775822292" watchObservedRunningTime="2026-09-08 00:33:12.43387654 +0000 UTC m=+26.777179562"
Sep 08 00:33:13.147368 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.147328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:13.147609 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.147328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:13.147609 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:13.147445    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:13.147609 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:13.147515    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:13.394913 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.394876    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc409628-ba63-48b6-ba42-5c86af963c50" containerID="0cd2b4026fbf01e05cdc769ec416fdfee88771886d2306084e201c8f69a6c2d1" exitCode=0
Sep 08 00:33:13.395079 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.394922    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerDied","Data":"0cd2b4026fbf01e05cdc769ec416fdfee88771886d2306084e201c8f69a6c2d1"}
Sep 08 00:33:13.398951 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.398935    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:33:13.399288 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.399268    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" event={"ID":"724f7c23-3ce3-45dd-932c-c8e0ce407e83","Type":"ContainerStarted","Data":"e76c14831ea3cfcb9581517ea1f2545c7254f276877b45719d8d274736155ab1"}
Sep 08 00:33:13.399388 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.399378    2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 00:33:13.399585 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.399572    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:33:13.414706 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.414687    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:33:13.467270 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:13.467221    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p" podStartSLOduration=8.629666052 podStartE2EDuration="27.467205097s" totalImagesPullingTime="18.837539045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:13.466830883 +0000 UTC m=+27.810133901" watchObservedRunningTime="2026-09-08 00:33:13.467205097 +0000 UTC m=+27.810508115"
Sep 08 00:33:14.146970 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.146921    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:14.147163 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:14.147075    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:14.183339 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.183302    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gmnq8"]
Sep 08 00:33:14.189336 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.189309    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fc6wt"]
Sep 08 00:33:14.189458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.189423    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:14.189527 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:14.189509    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:14.203688 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.203656    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hv2nm"]
Sep 08 00:33:14.203830 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.203778    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:14.203903 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:14.203880    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:14.403184 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.403083    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc409628-ba63-48b6-ba42-5c86af963c50" containerID="aad63947f0c7b1710172ccfa0ea03f936a2b35dd4c19e002604efcab23aa19fd" exitCode=0
Sep 08 00:33:14.403184 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.403147    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerDied","Data":"aad63947f0c7b1710172ccfa0ea03f936a2b35dd4c19e002604efcab23aa19fd"}
Sep 08 00:33:14.403376 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.403187    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:14.403485 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:14.403450    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:14.403609 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:14.403568    2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 00:33:15.407247 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:15.407213    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc409628-ba63-48b6-ba42-5c86af963c50" containerID="df10c587e9bf2231b146c22ff29366c16614c2b2edf57ff866245e0e6648c6ff" exitCode=0
Sep 08 00:33:15.407769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:15.407288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerDied","Data":"df10c587e9bf2231b146c22ff29366c16614c2b2edf57ff866245e0e6648c6ff"}
Sep 08 00:33:15.407769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:15.407385    2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 00:33:16.150316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:16.148915    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:16.150316 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:16.149269    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:16.150316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:16.149841    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:16.150316 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:16.149995    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:16.150316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:16.150050    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:16.150316 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:16.150205    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:16.929976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:16.929948    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:33:18.147459 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:18.147425    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:18.148032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:18.147425    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:18.148032 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:18.147561    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc6wt" podUID="7e5f18e2-434c-4068-9d88-d5e4f5f10a8f"
Sep 08 00:33:18.148032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:18.147437    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:18.148032 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:18.147659    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gmnq8" podUID="bd337a55-7af0-4b6c-98e9-a4980713cb79"
Sep 08 00:33:18.148032 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:18.147786    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hv2nm" podUID="e4e65bf2-ac1c-4438-a55f-e9984cae1b61"
Sep 08 00:33:18.916662 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:18.916622    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:18.916852 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:18.916713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:18.916852 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:18.916756    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:33:18.916852 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:18.916816    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:33:18.916852 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:18.916840    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret podName:7e5f18e2-434c-4068-9d88-d5e4f5f10a8f nodeName:}" failed. No retries permitted until 2026-09-08 00:33:50.916819065 +0000 UTC m=+65.260122093 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret") pod "global-pull-secret-syncer-fc6wt" (UID: "7e5f18e2-434c-4068-9d88-d5e4f5f10a8f") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:33:18.917077 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:18.916867    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:50.916849903 +0000 UTC m=+65.260152901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:33:19.017765 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.017737    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:19.017947 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.017925    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:33:19.018024 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.017957    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:33:19.018024 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.017970    2555 projected.go:196] Error preparing data for projected volume kube-api-access-5jdxn for pod openshift-network-diagnostics/network-check-target-hv2nm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:33:19.018122 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.018033    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn podName:e4e65bf2-ac1c-4438-a55f-e9984cae1b61 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:51.018014049 +0000 UTC m=+65.361317062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jdxn" (UniqueName: "kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn") pod "network-check-target-hv2nm" (UID: "e4e65bf2-ac1c-4438-a55f-e9984cae1b61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:33:19.483817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.483786    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-98.ec2.internal" event="NodeReady"
Sep 08 00:33:19.484225 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.483913    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 00:33:19.548549 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.548481    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b8f8c9778-pzvz5"]
Sep 08 00:33:19.643902 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.643870    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b8f8c9778-pzvz5"]
Sep 08 00:33:19.644040 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.643912    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d9jt5"]
Sep 08 00:33:19.644040 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.643953    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.648031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.648009    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 00:33:19.648264 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.648244    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ksxqj\""
Sep 08 00:33:19.648264 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.648244    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 00:33:19.648401 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.648334    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 00:33:19.653752 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.653735    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 00:33:19.655479 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.655465    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5d9c4"]
Sep 08 00:33:19.655583 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.655570    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.661716 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.661699    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 00:33:19.661894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.661878    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 00:33:19.661972 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.661925    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bs5xs\""
Sep 08 00:33:19.674513 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.674498    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d9jt5"]
Sep 08 00:33:19.674587 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.674523    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5d9c4"]
Sep 08 00:33:19.674587 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.674578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:19.677990 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.677969    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 00:33:19.678154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.678140    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 00:33:19.678227 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.678209    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2f7c4\""
Sep 08 00:33:19.678528 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.678513    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 00:33:19.723205 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-bound-sa-token\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.723305 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfgq\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-kube-api-access-rnfgq\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.723305 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723241    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-certificates\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.723305 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-image-registry-private-configuration\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.723416 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-trusted-ca\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.723416 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723391    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91a90b67-5af0-46df-b5dd-c2b7620c51ff-ca-trust-extracted\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.723416 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723411    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-installation-pull-secrets\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.723517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.723426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.808896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.808871    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ph5tr_5fea2dc6-69d9-4a32-8d6e-de47f938f884/konnectivity-agent/0.log" limit=1024
Sep 08 00:33:19.823657 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df6603-177a-429b-908b-b851cd4e4838-tmp-dir\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.823729 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823669    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-certificates\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.823729 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5xj\" (UniqueName: \"kubernetes.io/projected/77df6603-177a-429b-908b-b851cd4e4838-kube-api-access-cb5xj\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.823729 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-image-registry-private-configuration\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.823824 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77df6603-177a-429b-908b-b851cd4e4838-config-volume\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.823824 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-trusted-ca\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.823824 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvzz\" (UniqueName: \"kubernetes.io/projected/30a41bc2-2515-4913-8970-b29ef78f764b-kube-api-access-tjvzz\") pod \"ingress-canary-5d9c4\" (UID: \"30a41bc2-2515-4913-8970-b29ef78f764b\") " pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:19.823934 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91a90b67-5af0-46df-b5dd-c2b7620c51ff-ca-trust-extracted\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.823988 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.823967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-installation-pull-secrets\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.824094 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.824069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.824163 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.824145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-bound-sa-token\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.824222 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.824184    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfgq\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-kube-api-access-rnfgq\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.824222 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.824196    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 00:33:19.824222 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.824218    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8f8c9778-pzvz5: secret "image-registry-tls" not found
Sep 08 00:33:19.824357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.824223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.824357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.824239    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91a90b67-5af0-46df-b5dd-c2b7620c51ff-ca-trust-extracted\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.824357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.824262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30a41bc2-2515-4913-8970-b29ef78f764b-cert\") pod \"ingress-canary-5d9c4\" (UID: \"30a41bc2-2515-4913-8970-b29ef78f764b\") " pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:19.824357 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.824293    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls podName:91a90b67-5af0-46df-b5dd-c2b7620c51ff nodeName:}" failed. No retries permitted until 2026-09-08 00:33:20.324269258 +0000 UTC m=+34.667572268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls") pod "image-registry-7b8f8c9778-pzvz5" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff") : secret "image-registry-tls" not found
Sep 08 00:33:19.824357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.824331    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-certificates\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.825107 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.825087    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-trusted-ca\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.827569 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.827552    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-installation-pull-secrets\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.827569 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.827562    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-image-registry-private-configuration\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.836078 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.836058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-bound-sa-token\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.836499 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.836485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfgq\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-kube-api-access-rnfgq\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:19.906259 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.906239    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ph5tr_5fea2dc6-69d9-4a32-8d6e-de47f938f884/konnectivity-agent/0.log" limit=1024
Sep 08 00:33:19.925179 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77df6603-177a-429b-908b-b851cd4e4838-config-volume\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.925243 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvzz\" (UniqueName: \"kubernetes.io/projected/30a41bc2-2515-4913-8970-b29ef78f764b-kube-api-access-tjvzz\") pod \"ingress-canary-5d9c4\" (UID: \"30a41bc2-2515-4913-8970-b29ef78f764b\") " pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:19.925243 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.925316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30a41bc2-2515-4913-8970-b29ef78f764b-cert\") pod \"ingress-canary-5d9c4\" (UID: \"30a41bc2-2515-4913-8970-b29ef78f764b\") " pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:19.925316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925282    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df6603-177a-429b-908b-b851cd4e4838-tmp-dir\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.925316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925312    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5xj\" (UniqueName: \"kubernetes.io/projected/77df6603-177a-429b-908b-b851cd4e4838-kube-api-access-cb5xj\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.925455 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.925320    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:33:19.925455 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:19.925379    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls podName:77df6603-177a-429b-908b-b851cd4e4838 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:20.425362133 +0000 UTC m=+34.768665140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls") pod "dns-default-d9jt5" (UID: "77df6603-177a-429b-908b-b851cd4e4838") : secret "dns-default-metrics-tls" not found
Sep 08 00:33:19.925653 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925628    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77df6603-177a-429b-908b-b851cd4e4838-tmp-dir\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.925726 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.925700    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77df6603-177a-429b-908b-b851cd4e4838-config-volume\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.927337 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.927317    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30a41bc2-2515-4913-8970-b29ef78f764b-cert\") pod \"ingress-canary-5d9c4\" (UID: \"30a41bc2-2515-4913-8970-b29ef78f764b\") " pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:19.935217 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.935199    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5xj\" (UniqueName: \"kubernetes.io/projected/77df6603-177a-429b-908b-b851cd4e4838-kube-api-access-cb5xj\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:19.935873 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.935857    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvzz\" (UniqueName: \"kubernetes.io/projected/30a41bc2-2515-4913-8970-b29ef78f764b-kube-api-access-tjvzz\") pod \"ingress-canary-5d9c4\" (UID: \"30a41bc2-2515-4913-8970-b29ef78f764b\") " pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:19.982231 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:19.982055    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5d9c4"
Sep 08 00:33:20.147309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.147163    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:20.147403 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.147197    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:20.147639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.147222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:20.151441 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.151419    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 00:33:20.151539 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.151446    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wspvj\""
Sep 08 00:33:20.151656 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.151630    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 00:33:20.151786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.151705    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 00:33:20.151786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.151727    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 00:33:20.151786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.151767    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xvx4\""
Sep 08 00:33:20.294278 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.294252    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5d9c4"]
Sep 08 00:33:20.298054 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:33:20.298023    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a41bc2_2515_4913_8970_b29ef78f764b.slice/crio-d70608a7313c59b67ecb7b296de61a3d2e182e53cca70bf698ff38dabbc8422e WatchSource:0}: Error finding container d70608a7313c59b67ecb7b296de61a3d2e182e53cca70bf698ff38dabbc8422e: Status 404 returned error can't find the container with id d70608a7313c59b67ecb7b296de61a3d2e182e53cca70bf698ff38dabbc8422e
Sep 08 00:33:20.329351 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.329325    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:20.329519 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:20.329504    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 00:33:20.329562 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:20.329522    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8f8c9778-pzvz5: secret "image-registry-tls" not found
Sep 08 00:33:20.329615 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:20.329565    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls podName:91a90b67-5af0-46df-b5dd-c2b7620c51ff nodeName:}" failed. No retries permitted until 2026-09-08 00:33:21.329552978 +0000 UTC m=+35.672855976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls") pod "image-registry-7b8f8c9778-pzvz5" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff") : secret "image-registry-tls" not found
Sep 08 00:33:20.422580 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.422555    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc409628-ba63-48b6-ba42-5c86af963c50" containerID="76f040b75fd6d3af72fcbc0e2d667b6f06069a0c1580349d31c60a1b380f8303" exitCode=0
Sep 08 00:33:20.422691 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.422651    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerDied","Data":"76f040b75fd6d3af72fcbc0e2d667b6f06069a0c1580349d31c60a1b380f8303"}
Sep 08 00:33:20.423875 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.423850    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5d9c4" event={"ID":"30a41bc2-2515-4913-8970-b29ef78f764b","Type":"ContainerStarted","Data":"d70608a7313c59b67ecb7b296de61a3d2e182e53cca70bf698ff38dabbc8422e"}
Sep 08 00:33:20.429760 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:20.429735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:20.429854 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:20.429838    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:33:20.429901 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:20.429892    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls podName:77df6603-177a-429b-908b-b851cd4e4838 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:21.429875595 +0000 UTC m=+35.773178592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls") pod "dns-default-d9jt5" (UID: "77df6603-177a-429b-908b-b851cd4e4838") : secret "dns-default-metrics-tls" not found
Sep 08 00:33:21.336887 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:21.336854    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:21.337306 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:21.336980    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 00:33:21.337306 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:21.336995    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8f8c9778-pzvz5: secret "image-registry-tls" not found
Sep 08 00:33:21.337306 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:21.337057    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls podName:91a90b67-5af0-46df-b5dd-c2b7620c51ff nodeName:}" failed. No retries permitted until 2026-09-08 00:33:23.337038765 +0000 UTC m=+37.680341767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls") pod "image-registry-7b8f8c9778-pzvz5" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff") : secret "image-registry-tls" not found
Sep 08 00:33:21.428923 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:21.428884    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc409628-ba63-48b6-ba42-5c86af963c50" containerID="1f71b46545b2dfcc776bcf11c367355f54335a3fc3ef7798b1aa3b0531fe3a6b" exitCode=0
Sep 08 00:33:21.429058 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:21.428959    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerDied","Data":"1f71b46545b2dfcc776bcf11c367355f54335a3fc3ef7798b1aa3b0531fe3a6b"}
Sep 08 00:33:21.437144 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:21.437127    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:21.437294 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:21.437277    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:33:21.437359 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:21.437343    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls podName:77df6603-177a-429b-908b-b851cd4e4838 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:23.437323337 +0000 UTC m=+37.780626334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls") pod "dns-default-d9jt5" (UID: "77df6603-177a-429b-908b-b851cd4e4838") : secret "dns-default-metrics-tls" not found
Sep 08 00:33:22.436386 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:22.436318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" event={"ID":"dc409628-ba63-48b6-ba42-5c86af963c50","Type":"ContainerStarted","Data":"796be85083c2190275662d914e5a481835e19e70b821a28d22e440bd1703db11"}
Sep 08 00:33:22.520646 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:22.520571    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m6nxh" podStartSLOduration=6.754894659 podStartE2EDuration="36.520548847s" totalImagesPullingTime="29.765654188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:22.519522259 +0000 UTC m=+36.862825279" watchObservedRunningTime="2026-09-08 00:33:22.520548847 +0000 UTC m=+36.863851862"
Sep 08 00:33:23.350711 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:23.350687    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:23.350839 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:23.350821    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 00:33:23.350883 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:23.350841    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8f8c9778-pzvz5: secret "image-registry-tls" not found
Sep 08 00:33:23.350912 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:23.350891    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls podName:91a90b67-5af0-46df-b5dd-c2b7620c51ff nodeName:}" failed. No retries permitted until 2026-09-08 00:33:27.35087598 +0000 UTC m=+41.694178977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls") pod "image-registry-7b8f8c9778-pzvz5" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff") : secret "image-registry-tls" not found
Sep 08 00:33:23.452011 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:23.451986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:23.452358 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:23.452121    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:33:23.452358 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:23.452176    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls podName:77df6603-177a-429b-908b-b851cd4e4838 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:27.452161323 +0000 UTC m=+41.795464324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls") pod "dns-default-d9jt5" (UID: "77df6603-177a-429b-908b-b851cd4e4838") : secret "dns-default-metrics-tls" not found
Sep 08 00:33:24.442692 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:24.442657    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5d9c4" event={"ID":"30a41bc2-2515-4913-8970-b29ef78f764b","Type":"ContainerStarted","Data":"8849e7153b360eb13b493d71c0e5d09d45054e11aa079399973a35dd0891767e"}
Sep 08 00:33:24.471398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:24.471354    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5d9c4" podStartSLOduration=2.424967572 podStartE2EDuration="5.471340032s" totalImagesPullingTime="3.04637246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:24.470945099 +0000 UTC m=+38.814248122" watchObservedRunningTime="2026-09-08 00:33:24.471340032 +0000 UTC m=+38.814643068"
Sep 08 00:33:27.381234 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:27.381189    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:27.381764 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:27.381407    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 00:33:27.381764 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:27.381434    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8f8c9778-pzvz5: secret "image-registry-tls" not found
Sep 08 00:33:27.381764 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:27.381511    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls podName:91a90b67-5af0-46df-b5dd-c2b7620c51ff nodeName:}" failed. No retries permitted until 2026-09-08 00:33:35.381488233 +0000 UTC m=+49.724791237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls") pod "image-registry-7b8f8c9778-pzvz5" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff") : secret "image-registry-tls" not found
Sep 08 00:33:27.481987 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:27.481938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:27.482140 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:27.482090    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:33:27.482215 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:27.482157    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls podName:77df6603-177a-429b-908b-b851cd4e4838 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:35.482142308 +0000 UTC m=+49.825445304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls") pod "dns-default-d9jt5" (UID: "77df6603-177a-429b-908b-b851cd4e4838") : secret "dns-default-metrics-tls" not found
Sep 08 00:33:35.439455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:35.439415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:35.439914 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:35.439559    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 00:33:35.439914 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:35.439576    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8f8c9778-pzvz5: secret "image-registry-tls" not found
Sep 08 00:33:35.439914 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:35.439654    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls podName:91a90b67-5af0-46df-b5dd-c2b7620c51ff nodeName:}" failed. No retries permitted until 2026-09-08 00:33:51.439634636 +0000 UTC m=+65.782937653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls") pod "image-registry-7b8f8c9778-pzvz5" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff") : secret "image-registry-tls" not found
Sep 08 00:33:35.540123 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:35.540095    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:35.540251 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:35.540216    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:33:35.540288 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:35.540261    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls podName:77df6603-177a-429b-908b-b851cd4e4838 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:51.540248458 +0000 UTC m=+65.883551455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls") pod "dns-default-d9jt5" (UID: "77df6603-177a-429b-908b-b851cd4e4838") : secret "dns-default-metrics-tls" not found
Sep 08 00:33:46.941570 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:46.941539    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx52p"
Sep 08 00:33:50.947455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:50.947420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:50.947891 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:50.947480    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:33:50.950399 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:50.950376    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 00:33:50.950507 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:50.950383    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 00:33:50.958464 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:50.958441    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 00:33:50.958550 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:50.958510    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs podName:bd337a55-7af0-4b6c-98e9-a4980713cb79 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:54.958488179 +0000 UTC m=+129.301791190 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs") pod "network-metrics-daemon-gmnq8" (UID: "bd337a55-7af0-4b6c-98e9-a4980713cb79") : secret "metrics-daemon-secret" not found
Sep 08 00:33:50.961660 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:50.961627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e5f18e2-434c-4068-9d88-d5e4f5f10a8f-original-pull-secret\") pod \"global-pull-secret-syncer-fc6wt\" (UID: \"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f\") " pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:51.047748 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.047723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:51.050602 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.050581    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 00:33:51.060805 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.060786    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 00:33:51.066469 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.066458    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc6wt"
Sep 08 00:33:51.070705 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.070689    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdxn\" (UniqueName: \"kubernetes.io/projected/e4e65bf2-ac1c-4438-a55f-e9984cae1b61-kube-api-access-5jdxn\") pod \"network-check-target-hv2nm\" (UID: \"e4e65bf2-ac1c-4438-a55f-e9984cae1b61\") " pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:51.362662 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.362615    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wspvj\""
Sep 08 00:33:51.365410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.365382    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fc6wt"]
Sep 08 00:33:51.368509 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:33:51.368483    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f18e2_434c_4068_9d88_d5e4f5f10a8f.slice/crio-b3af8d155c4bb78d0d217f145e738a3a9a346d62a0ed7177c7833b126c83d224 WatchSource:0}: Error finding container b3af8d155c4bb78d0d217f145e738a3a9a346d62a0ed7177c7833b126c83d224: Status 404 returned error can't find the container with id b3af8d155c4bb78d0d217f145e738a3a9a346d62a0ed7177c7833b126c83d224
Sep 08 00:33:51.370401 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.370377    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:51.450243 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.450205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:33:51.450427 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:51.450340    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 00:33:51.450427 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:51.450362    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b8f8c9778-pzvz5: secret "image-registry-tls" not found
Sep 08 00:33:51.450536 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:51.450429    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls podName:91a90b67-5af0-46df-b5dd-c2b7620c51ff nodeName:}" failed. No retries permitted until 2026-09-08 00:34:23.450409997 +0000 UTC m=+97.793713010 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls") pod "image-registry-7b8f8c9778-pzvz5" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff") : secret "image-registry-tls" not found
Sep 08 00:33:51.509544 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.509509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fc6wt" event={"ID":"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f","Type":"ContainerStarted","Data":"b3af8d155c4bb78d0d217f145e738a3a9a346d62a0ed7177c7833b126c83d224"}
Sep 08 00:33:51.551155 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.551130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:33:51.551260 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:51.551248    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:33:51.551304 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:51.551298    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls podName:77df6603-177a-429b-908b-b851cd4e4838 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:23.551285007 +0000 UTC m=+97.894588004 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls") pod "dns-default-d9jt5" (UID: "77df6603-177a-429b-908b-b851cd4e4838") : secret "dns-default-metrics-tls" not found
Sep 08 00:33:51.638706 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:51.638638    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hv2nm"]
Sep 08 00:33:51.641516 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:33:51.641489    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e65bf2_ac1c_4438_a55f_e9984cae1b61.slice/crio-b7676419d9416eecefdef7f5d6cc535f73a6118f6b02bf2571f2a2a5de218384 WatchSource:0}: Error finding container b7676419d9416eecefdef7f5d6cc535f73a6118f6b02bf2571f2a2a5de218384: Status 404 returned error can't find the container with id b7676419d9416eecefdef7f5d6cc535f73a6118f6b02bf2571f2a2a5de218384
Sep 08 00:33:52.513899 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:52.513860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hv2nm" event={"ID":"e4e65bf2-ac1c-4438-a55f-e9984cae1b61","Type":"ContainerStarted","Data":"b7676419d9416eecefdef7f5d6cc535f73a6118f6b02bf2571f2a2a5de218384"}
Sep 08 00:33:56.526094 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:56.526053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fc6wt" event={"ID":"7e5f18e2-434c-4068-9d88-d5e4f5f10a8f","Type":"ContainerStarted","Data":"204b90d0f8931cd7d91f59ebd219ded2d9a7d1a7f8104e96bbdd406c2e1458f6"}
Sep 08 00:33:56.527741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:56.527716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hv2nm" event={"ID":"e4e65bf2-ac1c-4438-a55f-e9984cae1b61","Type":"ContainerStarted","Data":"9cdf2ca95676d241b7567269045e3c49ba471e5aaefdfb47bf52c3473f156f8e"}
Sep 08 00:33:56.527871 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:56.527851    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:33:56.541126 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:56.541074    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fc6wt" podStartSLOduration=65.546633286 podStartE2EDuration="1m10.541060099s" totalImagesPullingTime="4.994426813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:56.540446246 +0000 UTC m=+70.883749263" watchObservedRunningTime="2026-09-08 00:33:56.541060099 +0000 UTC m=+70.884363118"
Sep 08 00:33:56.558740 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:56.558693    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hv2nm" podStartSLOduration=65.840700167 podStartE2EDuration="1m10.558680869s" totalImagesPullingTime="4.717980702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:56.557936926 +0000 UTC m=+70.901239945" watchObservedRunningTime="2026-09-08 00:33:56.558680869 +0000 UTC m=+70.901983882"
Sep 08 00:33:59.069338 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.069302    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-q72jj"]
Sep 08 00:33:59.071077 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.071061    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:33:59.074300 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.074270    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4wfvl\""
Sep 08 00:33:59.074436 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.074334    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 00:33:59.075318 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.075301    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 00:33:59.085185 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.085162    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-q72jj"]
Sep 08 00:33:59.106188 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.106158    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:33:59.106311 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.106246    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8ab397c-8640-41f7-a47b-9a9a796412b6-nginx-conf\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:33:59.207242 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.207213    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8ab397c-8640-41f7-a47b-9a9a796412b6-nginx-conf\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:33:59.207355 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.207275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:33:59.207415 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.207379    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 00:33:59.207468 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.207430    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert podName:a8ab397c-8640-41f7-a47b-9a9a796412b6 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:59.707414025 +0000 UTC m=+74.050717040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q72jj" (UID: "a8ab397c-8640-41f7-a47b-9a9a796412b6") : secret "networking-console-plugin-cert" not found
Sep 08 00:33:59.207924 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.207901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8ab397c-8640-41f7-a47b-9a9a796412b6-nginx-conf\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:33:59.263248 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.263225    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"]
Sep 08 00:33:59.264912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.264898    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.267662 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.267639    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 00:33:59.267809 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.267790    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 00:33:59.267881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.267799    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 00:33:59.268094 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.268077    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7l4cf\""
Sep 08 00:33:59.269356 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.269339    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 00:33:59.276408 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.276384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"]
Sep 08 00:33:59.307862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.307839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.307945 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.307877    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rvh\" (UniqueName: \"kubernetes.io/projected/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-kube-api-access-z4rvh\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.307945 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.307914    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.408294 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.408246    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.408294 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.408279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rvh\" (UniqueName: \"kubernetes.io/projected/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-kube-api-access-z4rvh\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.408374 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.408314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.408405 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.408377    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 00:33:59.408441 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.408434    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls podName:b87e8aca-4a77-4c30-b7ba-bdda1c45ad50 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:59.908416035 +0000 UTC m=+74.251719036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8mpj2" (UID: "b87e8aca-4a77-4c30-b7ba-bdda1c45ad50") : secret "cluster-monitoring-operator-tls" not found
Sep 08 00:33:59.408948 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.408928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.418404 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.418377    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rvh\" (UniqueName: \"kubernetes.io/projected/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-kube-api-access-z4rvh\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.709761 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.709679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:33:59.709896 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.709799    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 00:33:59.709896 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.709849    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert podName:a8ab397c-8640-41f7-a47b-9a9a796412b6 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:00.70983626 +0000 UTC m=+75.053139257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q72jj" (UID: "a8ab397c-8640-41f7-a47b-9a9a796412b6") : secret "networking-console-plugin-cert" not found
Sep 08 00:33:59.911642 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:33:59.911613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:33:59.911823 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.911785    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 00:33:59.911890 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:33:59.911864    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls podName:b87e8aca-4a77-4c30-b7ba-bdda1c45ad50 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:00.911842216 +0000 UTC m=+75.255145214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8mpj2" (UID: "b87e8aca-4a77-4c30-b7ba-bdda1c45ad50") : secret "cluster-monitoring-operator-tls" not found
Sep 08 00:34:00.716899 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:00.716852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:34:00.717307 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:00.716995    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 00:34:00.717307 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:00.717055    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert podName:a8ab397c-8640-41f7-a47b-9a9a796412b6 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:02.717040788 +0000 UTC m=+77.060343785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q72jj" (UID: "a8ab397c-8640-41f7-a47b-9a9a796412b6") : secret "networking-console-plugin-cert" not found
Sep 08 00:34:00.918677 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:00.918634    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:34:00.918871 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:00.918785    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 00:34:00.918871 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:00.918866    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls podName:b87e8aca-4a77-4c30-b7ba-bdda1c45ad50 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:02.918846358 +0000 UTC m=+77.262149369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8mpj2" (UID: "b87e8aca-4a77-4c30-b7ba-bdda1c45ad50") : secret "cluster-monitoring-operator-tls" not found
Sep 08 00:34:02.731717 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:02.731670    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:34:02.732190 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:02.731800    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 00:34:02.732190 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:02.731852    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert podName:a8ab397c-8640-41f7-a47b-9a9a796412b6 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:06.731840211 +0000 UTC m=+81.075143207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q72jj" (UID: "a8ab397c-8640-41f7-a47b-9a9a796412b6") : secret "networking-console-plugin-cert" not found
Sep 08 00:34:02.932788 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:02.932745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:34:02.932965 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:02.932872    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 00:34:02.932965 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:02.932927    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls podName:b87e8aca-4a77-4c30-b7ba-bdda1c45ad50 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:06.932913628 +0000 UTC m=+81.276216638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8mpj2" (UID: "b87e8aca-4a77-4c30-b7ba-bdda1c45ad50") : secret "cluster-monitoring-operator-tls" not found
Sep 08 00:34:03.352443 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.352409    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j47nd"]
Sep 08 00:34:03.354668 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.354653    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.358234 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.358182    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 00:34:03.358380 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.358190    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 00:34:03.358380 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.358369    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xt6cx\""
Sep 08 00:34:03.358531 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.358482    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 00:34:03.359335 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.359318    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 00:34:03.365859 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.365837    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j47nd"]
Sep 08 00:34:03.436409 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.436378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7w5b\" (UniqueName: \"kubernetes.io/projected/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-api-access-k7w5b\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.436560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.436427    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-crio-socket\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.436560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.436529    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-data-volume\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.436560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.436554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.436686 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.436573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.537031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.537000    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7w5b\" (UniqueName: \"kubernetes.io/projected/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-api-access-k7w5b\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.537180 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.537050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-crio-socket\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.537180 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.537114    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-data-volume\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.537180 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.537130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.537180 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.537150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.537380 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.537191    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-crio-socket\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.537380 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:03.537244    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:34:03.537380 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:03.537328    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:04.03730679 +0000 UTC m=+78.380609805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : configmap "kube-rbac-proxy" not found
Sep 08 00:34:03.537380 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:03.537247    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:03.537562 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:03.537444    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:04.037425914 +0000 UTC m=+78.380728930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:03.537562 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.537491    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-data-volume\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:03.548482 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:03.548451    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7w5b\" (UniqueName: \"kubernetes.io/projected/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-api-access-k7w5b\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:04.041479 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.041426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:04.041479 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.041477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:04.041936 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:04.041551    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:34:04.041936 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:04.041642    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:05.041626597 +0000 UTC m=+79.384929594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : configmap "kube-rbac-proxy" not found
Sep 08 00:34:04.041936 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:04.041657    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:04.041936 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:04.041760    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:05.041748064 +0000 UTC m=+79.385051061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:04.700005 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.699976    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"]
Sep 08 00:34:04.702330 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.702312    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"
Sep 08 00:34:04.708707 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.708684    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 00:34:04.709935 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.709917    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5w7dl\""
Sep 08 00:34:04.710041 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.709943    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 00:34:04.719609 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.719566    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"]
Sep 08 00:34:04.747890 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.747861    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxwh\" (UniqueName: \"kubernetes.io/projected/3a95aabb-fe18-4514-8892-4d43ed3dda07-kube-api-access-msxwh\") pod \"migrator-fdbb8fdff-djv6g\" (UID: \"3a95aabb-fe18-4514-8892-4d43ed3dda07\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"
Sep 08 00:34:04.848214 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.848180    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxwh\" (UniqueName: \"kubernetes.io/projected/3a95aabb-fe18-4514-8892-4d43ed3dda07-kube-api-access-msxwh\") pod \"migrator-fdbb8fdff-djv6g\" (UID: \"3a95aabb-fe18-4514-8892-4d43ed3dda07\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"
Sep 08 00:34:04.857653 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:04.857622    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxwh\" (UniqueName: \"kubernetes.io/projected/3a95aabb-fe18-4514-8892-4d43ed3dda07-kube-api-access-msxwh\") pod \"migrator-fdbb8fdff-djv6g\" (UID: \"3a95aabb-fe18-4514-8892-4d43ed3dda07\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"
Sep 08 00:34:05.010853 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:05.010775    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"
Sep 08 00:34:05.050340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:05.050308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:05.050668 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:05.050347    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:05.050668 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:05.050440    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:34:05.050668 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:05.050496    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:07.050482562 +0000 UTC m=+81.393785564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : configmap "kube-rbac-proxy" not found
Sep 08 00:34:05.050668 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:05.050542    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:05.050668 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:05.050633    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:07.050617599 +0000 UTC m=+81.393920612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:05.104090 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:05.104066    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bcfgd_3460b5c1-7a31-4b5a-83df-5ff3ea4eac70/dns-node-resolver/0.log"
Sep 08 00:34:05.278893 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:05.278828    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g"]
Sep 08 00:34:05.281831 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:05.281806    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a95aabb_fe18_4514_8892_4d43ed3dda07.slice/crio-bd499c12e804127c2bbf5ac45e97058cb69243742c0646a5d6f265cd3bf4db3c WatchSource:0}: Error finding container bd499c12e804127c2bbf5ac45e97058cb69243742c0646a5d6f265cd3bf4db3c: Status 404 returned error can't find the container with id bd499c12e804127c2bbf5ac45e97058cb69243742c0646a5d6f265cd3bf4db3c
Sep 08 00:34:05.551346 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:05.551311    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g" event={"ID":"3a95aabb-fe18-4514-8892-4d43ed3dda07","Type":"ContainerStarted","Data":"bd499c12e804127c2bbf5ac45e97058cb69243742c0646a5d6f265cd3bf4db3c"}
Sep 08 00:34:06.303691 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:06.303654    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgjp4_9d01212a-e314-4028-a478-4fd6332e552d/node-ca/0.log"
Sep 08 00:34:06.557061 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:06.556977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g" event={"ID":"3a95aabb-fe18-4514-8892-4d43ed3dda07","Type":"ContainerStarted","Data":"d8b013ab3de576d715d171e40371bcedb2aed30c911f2307c7ca0fed2ea9d8a3"}
Sep 08 00:34:06.557061 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:06.557031    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g" event={"ID":"3a95aabb-fe18-4514-8892-4d43ed3dda07","Type":"ContainerStarted","Data":"46fb7f6fb8edeb67856710c442d93e0708188c135009ff3f1a22f2cb6723ed37"}
Sep 08 00:34:06.577425 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:06.577373    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-djv6g" podStartSLOduration=1.458652978 podStartE2EDuration="2.577358825s" totalImagesPullingTime="1.118705847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:06.577319146 +0000 UTC m=+80.920622166" watchObservedRunningTime="2026-09-08 00:34:06.577358825 +0000 UTC m=+80.920661846"
Sep 08 00:34:06.763710 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:06.763672    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:34:06.763849 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:06.763823    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 00:34:06.763904 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:06.763890    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert podName:a8ab397c-8640-41f7-a47b-9a9a796412b6 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:14.763873007 +0000 UTC m=+89.107176018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q72jj" (UID: "a8ab397c-8640-41f7-a47b-9a9a796412b6") : secret "networking-console-plugin-cert" not found
Sep 08 00:34:06.904548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:06.904463    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5d9c4_30a41bc2-2515-4913-8970-b29ef78f764b/serve-healthcheck-canary/0.log"
Sep 08 00:34:06.966245 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:06.966185    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:34:06.966395 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:06.966304    2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 00:34:06.966395 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:06.966358    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls podName:b87e8aca-4a77-4c30-b7ba-bdda1c45ad50 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:14.966345424 +0000 UTC m=+89.309648426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8mpj2" (UID: "b87e8aca-4a77-4c30-b7ba-bdda1c45ad50") : secret "cluster-monitoring-operator-tls" not found
Sep 08 00:34:07.067362 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:07.067329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:07.067362 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:07.067366    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:07.067559 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:07.067448    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:34:07.067559 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:07.067495    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:07.067559 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:07.067511    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:11.067496583 +0000 UTC m=+85.410799590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : configmap "kube-rbac-proxy" not found
Sep 08 00:34:07.067559 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:07.067530    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:11.067520481 +0000 UTC m=+85.410823479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:11.099174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:11.099058    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:11.099174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:11.099097    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:11.099567 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:11.099186    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:34:11.099567 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:11.099334    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:19.099311551 +0000 UTC m=+93.442614548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : configmap "kube-rbac-proxy" not found
Sep 08 00:34:11.099567 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:11.099344    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:11.099567 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:11.099401    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls podName:b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:19.099384954 +0000 UTC m=+93.442687972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j47nd" (UID: "b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:34:14.830982 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:14.830922    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:34:14.831407 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:14.831060    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 00:34:14.831407 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:14.831123    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert podName:a8ab397c-8640-41f7-a47b-9a9a796412b6 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:30.831109035 +0000 UTC m=+105.174412032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-q72jj" (UID: "a8ab397c-8640-41f7-a47b-9a9a796412b6") : secret "networking-console-plugin-cert" not found
Sep 08 00:34:15.032899 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:15.032856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:34:15.035161 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:15.035141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b87e8aca-4a77-4c30-b7ba-bdda1c45ad50-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8mpj2\" (UID: \"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:34:15.173281 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:15.173200    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"
Sep 08 00:34:15.444219 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:15.444147    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2"]
Sep 08 00:34:15.447294 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:15.447270    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87e8aca_4a77_4c30_b7ba_bdda1c45ad50.slice/crio-31821ba3487b1b3e403df2a7e036d7b623c124d182119a7fcdd4303cc71b47bf WatchSource:0}: Error finding container 31821ba3487b1b3e403df2a7e036d7b623c124d182119a7fcdd4303cc71b47bf: Status 404 returned error can't find the container with id 31821ba3487b1b3e403df2a7e036d7b623c124d182119a7fcdd4303cc71b47bf
Sep 08 00:34:15.585103 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:15.585072    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2" event={"ID":"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50","Type":"ContainerStarted","Data":"31821ba3487b1b3e403df2a7e036d7b623c124d182119a7fcdd4303cc71b47bf"}
Sep 08 00:34:17.592768 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:17.592727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2" event={"ID":"b87e8aca-4a77-4c30-b7ba-bdda1c45ad50","Type":"ContainerStarted","Data":"30c1c9806ef7b8c95050ed1785b6a478445861bc4a240f3336b809a6734bf6fa"}
Sep 08 00:34:17.612556 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:17.612509    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8mpj2" podStartSLOduration=16.917561515 podStartE2EDuration="18.612494221s" totalImagesPullingTime="1.694932706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:17.612158202 +0000 UTC m=+91.955461220" watchObservedRunningTime="2026-09-08 00:34:17.612494221 +0000 UTC m=+91.955797240"
Sep 08 00:34:19.165841 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:19.165807    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:19.166214 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:19.165964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:19.166420 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:19.166402    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:19.168068 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:19.168049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j47nd\" (UID: \"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2\") " pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:19.262992 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:19.262966    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j47nd"
Sep 08 00:34:19.537271 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:19.537197    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j47nd"]
Sep 08 00:34:19.540373 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:19.540331    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91da9ec_a2b0_4c6d_990d_7cfd9a6960b2.slice/crio-fc09b2820fa5c17fb2baeeaf5031495a8ae4ebc26c23f65509a0eae1081f4d14 WatchSource:0}: Error finding container fc09b2820fa5c17fb2baeeaf5031495a8ae4ebc26c23f65509a0eae1081f4d14: Status 404 returned error can't find the container with id fc09b2820fa5c17fb2baeeaf5031495a8ae4ebc26c23f65509a0eae1081f4d14
Sep 08 00:34:19.601635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:19.601606    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j47nd" event={"ID":"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2","Type":"ContainerStarted","Data":"fc09b2820fa5c17fb2baeeaf5031495a8ae4ebc26c23f65509a0eae1081f4d14"}
Sep 08 00:34:20.606941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:20.606905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j47nd" event={"ID":"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2","Type":"ContainerStarted","Data":"f62106605828d1457ac7e2ced887c3a929250247bf238b1a013bd3a45d27a545"}
Sep 08 00:34:20.607291 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:20.606971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j47nd" event={"ID":"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2","Type":"ContainerStarted","Data":"e36c0c28a501dd3685333d646335047543019a89f581ade9b0770c4c72f00e79"}
Sep 08 00:34:22.617180 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:22.617143    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j47nd" event={"ID":"b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2","Type":"ContainerStarted","Data":"814b104e8d6c4d86b153df5283bcabba22e4227e3aa7b288a42a2a06b9f93691"}
Sep 08 00:34:22.637732 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:22.637682    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j47nd" podStartSLOduration=17.439758452 podStartE2EDuration="19.637665247s" totalImagesPullingTime="2.197906795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:34:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:22.63571701 +0000 UTC m=+96.979020028" watchObservedRunningTime="2026-09-08 00:34:22.637665247 +0000 UTC m=+96.980968265"
Sep 08 00:34:23.508488 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.508443    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:34:23.510835 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.510813    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"image-registry-7b8f8c9778-pzvz5\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") " pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:34:23.555672 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.555643    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ksxqj\""
Sep 08 00:34:23.562953 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.562923    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:34:23.609636 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.609478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:34:23.615270 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.612809    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77df6603-177a-429b-908b-b851cd4e4838-metrics-tls\") pod \"dns-default-d9jt5\" (UID: \"77df6603-177a-429b-908b-b851cd4e4838\") " pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:34:23.837979 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.837916    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b8f8c9778-pzvz5"]
Sep 08 00:34:23.840554 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:23.840532    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a90b67_5af0_46df_b5dd_c2b7620c51ff.slice/crio-2f3719ee7dc6cd477f1552dcd00c16383efad7c93f0ff5f4e78c1a18e3083416 WatchSource:0}: Error finding container 2f3719ee7dc6cd477f1552dcd00c16383efad7c93f0ff5f4e78c1a18e3083416: Status 404 returned error can't find the container with id 2f3719ee7dc6cd477f1552dcd00c16383efad7c93f0ff5f4e78c1a18e3083416
Sep 08 00:34:23.866224 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.866197    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bs5xs\""
Sep 08 00:34:23.874334 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:23.874312    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:34:24.144769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:24.144686    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d9jt5"]
Sep 08 00:34:24.148562 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:24.148529    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77df6603_177a_429b_908b_b851cd4e4838.slice/crio-0321570d3609049d22c8c6a88b960446d6a8add08277fa2adf5b294c9e7ad239 WatchSource:0}: Error finding container 0321570d3609049d22c8c6a88b960446d6a8add08277fa2adf5b294c9e7ad239: Status 404 returned error can't find the container with id 0321570d3609049d22c8c6a88b960446d6a8add08277fa2adf5b294c9e7ad239
Sep 08 00:34:24.626490 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:24.626449    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5" event={"ID":"91a90b67-5af0-46df-b5dd-c2b7620c51ff","Type":"ContainerStarted","Data":"5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90"}
Sep 08 00:34:24.626490 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:24.626517    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5" event={"ID":"91a90b67-5af0-46df-b5dd-c2b7620c51ff","Type":"ContainerStarted","Data":"2f3719ee7dc6cd477f1552dcd00c16383efad7c93f0ff5f4e78c1a18e3083416"}
Sep 08 00:34:24.626780 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:24.626543    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:34:24.627828 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:24.627800    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d9jt5" event={"ID":"77df6603-177a-429b-908b-b851cd4e4838","Type":"ContainerStarted","Data":"0321570d3609049d22c8c6a88b960446d6a8add08277fa2adf5b294c9e7ad239"}
Sep 08 00:34:24.647613 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:24.647514    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5" podStartSLOduration=98.64749528 podStartE2EDuration="1m38.64749528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:24.64681623 +0000 UTC m=+98.990119277" watchObservedRunningTime="2026-09-08 00:34:24.64749528 +0000 UTC m=+98.990798311"
Sep 08 00:34:26.636431 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:26.636390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d9jt5" event={"ID":"77df6603-177a-429b-908b-b851cd4e4838","Type":"ContainerStarted","Data":"14dc4b05df49099486824238e0516c4df85714060929d29a32c30b7f1a3082f7"}
Sep 08 00:34:26.636937 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:26.636484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d9jt5" event={"ID":"77df6603-177a-429b-908b-b851cd4e4838","Type":"ContainerStarted","Data":"359bbc1239c713e26a1a9bf0425df6d2917f77c46900aa0830be36b8ab506fdf"}
Sep 08 00:34:26.636937 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:26.636558    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:34:26.655661 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:26.655609    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d9jt5" podStartSLOduration=65.970133169 podStartE2EDuration="1m7.655576243s" totalImagesPullingTime="1.685443074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:26.654550257 +0000 UTC m=+100.997853275" watchObservedRunningTime="2026-09-08 00:34:26.655576243 +0000 UTC m=+100.998879261"
Sep 08 00:34:27.532517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:27.532485    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hv2nm"
Sep 08 00:34:29.505663 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.505626    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"]
Sep 08 00:34:29.536615 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.536566    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"]
Sep 08 00:34:29.536774 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.536691    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.539745 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.539721    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 00:34:29.539921 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.539721    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 00:34:29.541133 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.541112    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 00:34:29.541201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.541138    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 00:34:29.604607 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.604567    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"]
Sep 08 00:34:29.620407 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.620373    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-25kz4"]
Sep 08 00:34:29.620575 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.620512    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"
Sep 08 00:34:29.624678 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.624642    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bw92n\""
Sep 08 00:34:29.624678 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.624666    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 00:34:29.634396 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.634375    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-25kz4"]
Sep 08 00:34:29.634522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.634403    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"]
Sep 08 00:34:29.634522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.634498    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-25kz4"
Sep 08 00:34:29.638351 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.638336    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 00:34:29.638351 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.638343    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 00:34:29.638497 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.638373    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4mzf5\""
Sep 08 00:34:29.654661 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.654620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-tmp\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.654788 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.654697    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-klusterlet-config\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.654788 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.654716    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x56b\" (UniqueName: \"kubernetes.io/projected/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-kube-api-access-8x56b\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.755255 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.755226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-klusterlet-config\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.755255 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.755259    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtsm\" (UniqueName: \"kubernetes.io/projected/67ed3f60-57cd-4843-96e9-e81b35eae7c5-kube-api-access-pgtsm\") pod \"downloads-85d98c998f-25kz4\" (UID: \"67ed3f60-57cd-4843-96e9-e81b35eae7c5\") " pod="openshift-console/downloads-85d98c998f-25kz4"
Sep 08 00:34:29.755433 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.755304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x56b\" (UniqueName: \"kubernetes.io/projected/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-kube-api-access-8x56b\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.755433 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.755364    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fae4df9-76bb-45ab-a175-0f821d73db39-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-q8k4l\" (UID: \"5fae4df9-76bb-45ab-a175-0f821d73db39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"
Sep 08 00:34:29.755433 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.755405    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-tmp\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.755832 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.755788    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-tmp\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.757687 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.757671    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-klusterlet-config\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.803250 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.803220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x56b\" (UniqueName: \"kubernetes.io/projected/6f13342d-bdc6-4b2f-b466-5cc07fb9545a-kube-api-access-8x56b\") pod \"klusterlet-addon-workmgr-b44756db8-5l9wt\" (UID: \"6f13342d-bdc6-4b2f-b466-5cc07fb9545a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.845748 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.845724    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:29.856023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.855998    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fae4df9-76bb-45ab-a175-0f821d73db39-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-q8k4l\" (UID: \"5fae4df9-76bb-45ab-a175-0f821d73db39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"
Sep 08 00:34:29.856118 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.856054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtsm\" (UniqueName: \"kubernetes.io/projected/67ed3f60-57cd-4843-96e9-e81b35eae7c5-kube-api-access-pgtsm\") pod \"downloads-85d98c998f-25kz4\" (UID: \"67ed3f60-57cd-4843-96e9-e81b35eae7c5\") " pod="openshift-console/downloads-85d98c998f-25kz4"
Sep 08 00:34:29.858115 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.858093    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5fae4df9-76bb-45ab-a175-0f821d73db39-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-q8k4l\" (UID: \"5fae4df9-76bb-45ab-a175-0f821d73db39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"
Sep 08 00:34:29.865134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.865114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtsm\" (UniqueName: \"kubernetes.io/projected/67ed3f60-57cd-4843-96e9-e81b35eae7c5-kube-api-access-pgtsm\") pod \"downloads-85d98c998f-25kz4\" (UID: \"67ed3f60-57cd-4843-96e9-e81b35eae7c5\") " pod="openshift-console/downloads-85d98c998f-25kz4"
Sep 08 00:34:29.930938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.930911    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"
Sep 08 00:34:29.943055 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:29.943027    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-25kz4"
Sep 08 00:34:30.129967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.129939    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"]
Sep 08 00:34:30.132749 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:30.132726    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f13342d_bdc6_4b2f_b466_5cc07fb9545a.slice/crio-820b5edf84d76c727471f5152559feb2e93819dd34e629083fad83d25b911839 WatchSource:0}: Error finding container 820b5edf84d76c727471f5152559feb2e93819dd34e629083fad83d25b911839: Status 404 returned error can't find the container with id 820b5edf84d76c727471f5152559feb2e93819dd34e629083fad83d25b911839
Sep 08 00:34:30.221749 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.221725    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"]
Sep 08 00:34:30.224460 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:30.224433    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fae4df9_76bb_45ab_a175_0f821d73db39.slice/crio-64e9a708ae0103382a8a2f895c60d7465fa2de4812b98928c5a32675e568473c WatchSource:0}: Error finding container 64e9a708ae0103382a8a2f895c60d7465fa2de4812b98928c5a32675e568473c: Status 404 returned error can't find the container with id 64e9a708ae0103382a8a2f895c60d7465fa2de4812b98928c5a32675e568473c
Sep 08 00:34:30.254896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.254661    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-25kz4"]
Sep 08 00:34:30.650479 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.650438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l" event={"ID":"5fae4df9-76bb-45ab-a175-0f821d73db39","Type":"ContainerStarted","Data":"64e9a708ae0103382a8a2f895c60d7465fa2de4812b98928c5a32675e568473c"}
Sep 08 00:34:30.651539 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.651513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt" event={"ID":"6f13342d-bdc6-4b2f-b466-5cc07fb9545a","Type":"ContainerStarted","Data":"820b5edf84d76c727471f5152559feb2e93819dd34e629083fad83d25b911839"}
Sep 08 00:34:30.652474 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.652456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-25kz4" event={"ID":"67ed3f60-57cd-4843-96e9-e81b35eae7c5","Type":"ContainerStarted","Data":"593bf070275ead8b7f404a1bf4631fe3ab5808cd9fc601c5251759eedd9daff0"}
Sep 08 00:34:30.863461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.863423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:34:30.866101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.866078    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8ab397c-8640-41f7-a47b-9a9a796412b6-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-q72jj\" (UID: \"a8ab397c-8640-41f7-a47b-9a9a796412b6\") " pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:34:30.878987 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:30.878959    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj"
Sep 08 00:34:31.179736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:31.179615    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-q72jj"]
Sep 08 00:34:31.597881 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:31.597846    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ab397c_8640_41f7_a47b_9a9a796412b6.slice/crio-235b6826eedb6ff0b3901e04496e073b2c2f7598589ce5718cd33feb0fae8a9c WatchSource:0}: Error finding container 235b6826eedb6ff0b3901e04496e073b2c2f7598589ce5718cd33feb0fae8a9c: Status 404 returned error can't find the container with id 235b6826eedb6ff0b3901e04496e073b2c2f7598589ce5718cd33feb0fae8a9c
Sep 08 00:34:31.660238 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:31.659951    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj" event={"ID":"a8ab397c-8640-41f7-a47b-9a9a796412b6","Type":"ContainerStarted","Data":"235b6826eedb6ff0b3901e04496e073b2c2f7598589ce5718cd33feb0fae8a9c"}
Sep 08 00:34:32.667788 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:32.667729    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l" event={"ID":"5fae4df9-76bb-45ab-a175-0f821d73db39","Type":"ContainerStarted","Data":"e7d17dde4d013c4a1331cc2df2fff8277646f0bbe26fad35760ac82babd684c1"}
Sep 08 00:34:32.668468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:32.668441    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"
Sep 08 00:34:32.674845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:32.674811    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l"
Sep 08 00:34:32.690756 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:32.690630    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q8k4l" podStartSLOduration=2.296346674 podStartE2EDuration="3.690610232s" totalImagesPullingTime="1.394263558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:32.689622611 +0000 UTC m=+107.032925630" watchObservedRunningTime="2026-09-08 00:34:32.690610232 +0000 UTC m=+107.033913243"
Sep 08 00:34:35.682776 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:35.682731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj" event={"ID":"a8ab397c-8640-41f7-a47b-9a9a796412b6","Type":"ContainerStarted","Data":"6cfb4e2cec36a8503525540d6dd678005372aefabd8fd9e29bfa52e2a5645ff9"}
Sep 08 00:34:35.684677 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:35.684644    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt" event={"ID":"6f13342d-bdc6-4b2f-b466-5cc07fb9545a","Type":"ContainerStarted","Data":"30baa140a0e6cadc30db96c3adfd63edc74682e5cb785b56b49b13cc44432da4"}
Sep 08 00:34:35.684938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:35.684903    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:35.686974 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:35.686957    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt"
Sep 08 00:34:35.700409 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:35.700360    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-q72jj" podStartSLOduration=33.293961803 podStartE2EDuration="36.70034125s" totalImagesPullingTime="3.406379447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:35.699313359 +0000 UTC m=+110.042616379" watchObservedRunningTime="2026-09-08 00:34:35.70034125 +0000 UTC m=+110.043644270"
Sep 08 00:34:35.723542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:35.723482    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b44756db8-5l9wt" podStartSLOduration=1.831765083 podStartE2EDuration="6.723464869s" totalImagesPullingTime="4.891699786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:35.722538614 +0000 UTC m=+110.065841658" watchObservedRunningTime="2026-09-08 00:34:35.723464869 +0000 UTC m=+110.066767888"
Sep 08 00:34:36.641633 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:36.641579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d9jt5"
Sep 08 00:34:38.092288 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.092249    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"]
Sep 08 00:34:38.095543 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.095516    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.098612 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.098570    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 08 00:34:38.098769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.098657    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 08 00:34:38.099522 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.099500    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 00:34:38.099624 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.099501    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8jrnk\""
Sep 08 00:34:38.116826 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.116784    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"]
Sep 08 00:34:38.124075 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.124037    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.124280 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.124095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68fr\" (UniqueName: \"kubernetes.io/projected/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-kube-api-access-x68fr\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.124280 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.124165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.124405 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.124315    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.152077 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.152044    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pm9p9"]
Sep 08 00:34:38.155480 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.155456    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.158872 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.158694    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 00:34:38.158872 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.158730    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5q7dp\""
Sep 08 00:34:38.158872 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.158779    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 00:34:38.158872 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.158706    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 00:34:38.225740 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-tls\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.225936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-accelerators-collector-config\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.225936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.225936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225835    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.225936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-wtmp\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.225936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68fr\" (UniqueName: \"kubernetes.io/projected/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-kube-api-access-x68fr\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.225936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225900    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.226197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.225957    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx45c\" (UniqueName: \"kubernetes.io/projected/5ce59eab-6fe6-45da-b56e-5297ec1da3db-kube-api-access-tx45c\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.226197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.226035    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-sys\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.226197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.226084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.226197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.226119    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce59eab-6fe6-45da-b56e-5297ec1da3db-metrics-client-ca\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.226197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.226150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-textfile\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.226197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.226193    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-root\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.226482 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:38.226228    2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 08 00:34:38.226482 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:38.226315    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls podName:a0576d64-f3ba-4128-bca9-2ac8ec0190cd nodeName:}" failed. No retries permitted until 2026-09-08 00:34:38.726291627 +0000 UTC m=+113.069594624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-g2gms" (UID: "a0576d64-f3ba-4128-bca9-2ac8ec0190cd") : secret "openshift-state-metrics-tls" not found
Sep 08 00:34:38.226705 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.226680    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.228504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.228485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.246083 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.246049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68fr\" (UniqueName: \"kubernetes.io/projected/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-kube-api-access-x68fr\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.326729 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.326688    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.326912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.326739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx45c\" (UniqueName: \"kubernetes.io/projected/5ce59eab-6fe6-45da-b56e-5297ec1da3db-kube-api-access-tx45c\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.326912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.326791    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-sys\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.326912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.326835    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce59eab-6fe6-45da-b56e-5297ec1da3db-metrics-client-ca\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.326912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.326869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-textfile\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.326937    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-sys\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.326980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-root\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-tls\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-accelerators-collector-config\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327091    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-wtmp\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-root\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327441 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:38.327133    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 00:34:38.327441 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327203    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-textfile\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327441 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:38.327210    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-tls podName:5ce59eab-6fe6-45da-b56e-5297ec1da3db nodeName:}" failed. No retries permitted until 2026-09-08 00:34:38.827188841 +0000 UTC m=+113.170491843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-tls") pod "node-exporter-pm9p9" (UID: "5ce59eab-6fe6-45da-b56e-5297ec1da3db") : secret "node-exporter-tls" not found
Sep 08 00:34:38.327441 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327238    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-wtmp\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327615 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ce59eab-6fe6-45da-b56e-5297ec1da3db-metrics-client-ca\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.327734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.327713    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-accelerators-collector-config\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.329296 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.329269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.336950 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.336927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx45c\" (UniqueName: \"kubernetes.io/projected/5ce59eab-6fe6-45da-b56e-5297ec1da3db-kube-api-access-tx45c\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.730793 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.730758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:38.730972 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:38.730877    2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 08 00:34:38.730972 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:38.730942    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls podName:a0576d64-f3ba-4128-bca9-2ac8ec0190cd nodeName:}" failed. No retries permitted until 2026-09-08 00:34:39.730927403 +0000 UTC m=+114.074230400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-g2gms" (UID: "a0576d64-f3ba-4128-bca9-2ac8ec0190cd") : secret "openshift-state-metrics-tls" not found
Sep 08 00:34:38.831751 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.831715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-tls\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:38.834353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:38.834328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ce59eab-6fe6-45da-b56e-5297ec1da3db-node-exporter-tls\") pod \"node-exporter-pm9p9\" (UID: \"5ce59eab-6fe6-45da-b56e-5297ec1da3db\") " pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:39.066582 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:39.066542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pm9p9"
Sep 08 00:34:39.079913 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:39.079882    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce59eab_6fe6_45da_b56e_5297ec1da3db.slice/crio-a270a40a688633f9d76f46fdf0258e30953b312881a0721c7ac99e7661bbedea WatchSource:0}: Error finding container a270a40a688633f9d76f46fdf0258e30953b312881a0721c7ac99e7661bbedea: Status 404 returned error can't find the container with id a270a40a688633f9d76f46fdf0258e30953b312881a0721c7ac99e7661bbedea
Sep 08 00:34:39.703173 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:39.703128    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm9p9" event={"ID":"5ce59eab-6fe6-45da-b56e-5297ec1da3db","Type":"ContainerStarted","Data":"a270a40a688633f9d76f46fdf0258e30953b312881a0721c7ac99e7661bbedea"}
Sep 08 00:34:39.739860 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:39.739821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:39.743092 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:39.743036    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0576d64-f3ba-4128-bca9-2ac8ec0190cd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g2gms\" (UID: \"a0576d64-f3ba-4128-bca9-2ac8ec0190cd\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:39.906945 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:39.906908    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"
Sep 08 00:34:40.415283 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.415201    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms"]
Sep 08 00:34:40.419330 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:40.419293    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0576d64_f3ba_4128_bca9_2ac8ec0190cd.slice/crio-c4fa875d9143996fc8c17a86d3ea58b5125fb652bfcefeaa68262558a0cf1121 WatchSource:0}: Error finding container c4fa875d9143996fc8c17a86d3ea58b5125fb652bfcefeaa68262558a0cf1121: Status 404 returned error can't find the container with id c4fa875d9143996fc8c17a86d3ea58b5125fb652bfcefeaa68262558a0cf1121
Sep 08 00:34:40.558683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.558645    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57679cb458-2t89t"]
Sep 08 00:34:40.562287 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.562260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.565001 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.564973    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 00:34:40.565381 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.565351    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 00:34:40.565516 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.565385    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9ml86\""
Sep 08 00:34:40.565516 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.565451    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 00:34:40.566272 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.566252    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 00:34:40.566364 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.566285    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 00:34:40.571708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.571685    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 00:34:40.577542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.577519    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57679cb458-2t89t"]
Sep 08 00:34:40.649130 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.649078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-oauth-serving-cert\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.649130 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.649134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-oauth-config\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.649359 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.649169    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-trusted-ca-bundle\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.649359 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.649258    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-config\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.649359 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.649316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86s8j\" (UniqueName: \"kubernetes.io/projected/ae294dcf-ee38-4fb4-98fe-53be942da6f8-kube-api-access-86s8j\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.649499 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.649385    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-service-ca\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.649499 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.649416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-serving-cert\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.709256 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.709124    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms" event={"ID":"a0576d64-f3ba-4128-bca9-2ac8ec0190cd","Type":"ContainerStarted","Data":"a0b9bb16eaba33e1b9017b73802a71f5b12cec63a4da092f858da3ba45de23de"}
Sep 08 00:34:40.709256 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.709203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms" event={"ID":"a0576d64-f3ba-4128-bca9-2ac8ec0190cd","Type":"ContainerStarted","Data":"c4fa875d9143996fc8c17a86d3ea58b5125fb652bfcefeaa68262558a0cf1121"}
Sep 08 00:34:40.711143 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.711115    2555 generic.go:356] "Generic (PLEG): container finished" podID="5ce59eab-6fe6-45da-b56e-5297ec1da3db" containerID="c51b215f3572301f7775fcbbbec4bc50f3b5e6e71dd79e5bb9454875eae0fe97" exitCode=0
Sep 08 00:34:40.711264 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.711182    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm9p9" event={"ID":"5ce59eab-6fe6-45da-b56e-5297ec1da3db","Type":"ContainerDied","Data":"c51b215f3572301f7775fcbbbec4bc50f3b5e6e71dd79e5bb9454875eae0fe97"}
Sep 08 00:34:40.750584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.750544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86s8j\" (UniqueName: \"kubernetes.io/projected/ae294dcf-ee38-4fb4-98fe-53be942da6f8-kube-api-access-86s8j\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.750771 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.750632    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-service-ca\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.750771 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.750663    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-serving-cert\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.750771 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.750701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-oauth-serving-cert\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.750951 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.750926    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-oauth-config\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.752667 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.751053    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-trusted-ca-bundle\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.752667 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.751159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-config\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.752667 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.751473    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-service-ca\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.752667 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.752099    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-config\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.752667 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.752627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-oauth-serving-cert\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.752965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.752759    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-trusted-ca-bundle\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.753721 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.753699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-serving-cert\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.753908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.753891    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-oauth-config\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.761960 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.761936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86s8j\" (UniqueName: \"kubernetes.io/projected/ae294dcf-ee38-4fb4-98fe-53be942da6f8-kube-api-access-86s8j\") pod \"console-57679cb458-2t89t\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") " pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:40.873888 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:40.873846    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:34:41.136649 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.136615    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-656dc89847-bqg56"]
Sep 08 00:34:41.140569 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.140540    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.146001 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.145972    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 08 00:34:41.146547 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.146524    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 08 00:34:41.146689 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.146528    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 08 00:34:41.146755 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.146696    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mtnbg\""
Sep 08 00:34:41.146936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.146917    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7s3soh2o0e0ml\""
Sep 08 00:34:41.147030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.146940    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 08 00:34:41.147030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.146945    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 08 00:34:41.159275 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.159244    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-656dc89847-bqg56"]
Sep 08 00:34:41.255266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255222    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-tls\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.255266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.255501 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255296    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hpwp\" (UniqueName: \"kubernetes.io/projected/92e2354e-21c3-450f-a060-00fb2141c3a8-kube-api-access-6hpwp\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.255501 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.255501 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255423    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.255501 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255443    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.255694 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92e2354e-21c3-450f-a060-00fb2141c3a8-metrics-client-ca\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.255694 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.255542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-grpc-tls\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356403 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92e2354e-21c3-450f-a060-00fb2141c3a8-metrics-client-ca\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-grpc-tls\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-tls\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356488    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hpwp\" (UniqueName: \"kubernetes.io/projected/92e2354e-21c3-450f-a060-00fb2141c3a8-kube-api-access-6hpwp\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356575    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356925 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356647    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.356925 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.356674    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.357415 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.357332    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92e2354e-21c3-450f-a060-00fb2141c3a8-metrics-client-ca\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.359787 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.359716    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.359909 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.359887    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-grpc-tls\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.360169 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.360139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.360276 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.360240    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.360410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.360349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.360410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.360357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/92e2354e-21c3-450f-a060-00fb2141c3a8-secret-thanos-querier-tls\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.370725 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.370703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hpwp\" (UniqueName: \"kubernetes.io/projected/92e2354e-21c3-450f-a060-00fb2141c3a8-kube-api-access-6hpwp\") pod \"thanos-querier-656dc89847-bqg56\" (UID: \"92e2354e-21c3-450f-a060-00fb2141c3a8\") " pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:41.451754 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:41.451661    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:42.869800 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:42.869760    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"]
Sep 08 00:34:42.873033 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:42.873014    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:42.875868 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:42.875842    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 00:34:42.875973 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:42.875850    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mpl2c\""
Sep 08 00:34:42.894397 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:42.894372    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"]
Sep 08 00:34:42.971338 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:42.971304    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tln6\" (UniqueName: \"kubernetes.io/projected/2625f9b0-0053-4570-a88c-cc2172af60a5-kube-api-access-2tln6\") pod \"monitoring-plugin-85bbbbbf4-tdv7m\" (UID: \"2625f9b0-0053-4570-a88c-cc2172af60a5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:42.971493 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:42.971348    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2625f9b0-0053-4570-a88c-cc2172af60a5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tdv7m\" (UID: \"2625f9b0-0053-4570-a88c-cc2172af60a5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:43.072582 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.072547    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tln6\" (UniqueName: \"kubernetes.io/projected/2625f9b0-0053-4570-a88c-cc2172af60a5-kube-api-access-2tln6\") pod \"monitoring-plugin-85bbbbbf4-tdv7m\" (UID: \"2625f9b0-0053-4570-a88c-cc2172af60a5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:43.072781 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.072619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2625f9b0-0053-4570-a88c-cc2172af60a5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tdv7m\" (UID: \"2625f9b0-0053-4570-a88c-cc2172af60a5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:43.072781 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:43.072756    2555 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 08 00:34:43.072907 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:34:43.072827    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2625f9b0-0053-4570-a88c-cc2172af60a5-monitoring-plugin-cert podName:2625f9b0-0053-4570-a88c-cc2172af60a5 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:43.572806421 +0000 UTC m=+117.916109437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2625f9b0-0053-4570-a88c-cc2172af60a5-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-tdv7m" (UID: "2625f9b0-0053-4570-a88c-cc2172af60a5") : secret "monitoring-plugin-cert" not found
Sep 08 00:34:43.092647 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.092612    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tln6\" (UniqueName: \"kubernetes.io/projected/2625f9b0-0053-4570-a88c-cc2172af60a5-kube-api-access-2tln6\") pod \"monitoring-plugin-85bbbbbf4-tdv7m\" (UID: \"2625f9b0-0053-4570-a88c-cc2172af60a5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:43.416124 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.416091    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67fc864b6-ss9cn"]
Sep 08 00:34:43.419948 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.419926    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.423701 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.423668    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 08 00:34:43.423818 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.423802    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 08 00:34:43.423882 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.423830    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 08 00:34:43.423937 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.423927    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 08 00:34:43.424233 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.424214    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jwg77\""
Sep 08 00:34:43.424373 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.424352    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 08 00:34:43.429683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.429661    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 08 00:34:43.438693 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.438665    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67fc864b6-ss9cn"]
Sep 08 00:34:43.476722 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.476688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-secret-telemeter-client\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.476871 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.476742    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-metrics-client-ca\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.476871 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.476805    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-telemeter-client-tls\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.477015 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.476869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.477015 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.476938    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.477015 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.477001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-serving-certs-ca-bundle\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.477160 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.477032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhr7r\" (UniqueName: \"kubernetes.io/projected/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-kube-api-access-mhr7r\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.477160 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.477078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-federate-client-tls\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.577877 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.577843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.578018 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.577907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2625f9b0-0053-4570-a88c-cc2172af60a5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tdv7m\" (UID: \"2625f9b0-0053-4570-a88c-cc2172af60a5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:43.578018 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.577935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.578018 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.577968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-serving-certs-ca-bundle\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.578018 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.577992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhr7r\" (UniqueName: \"kubernetes.io/projected/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-kube-api-access-mhr7r\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.578733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.578282    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-federate-client-tls\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.578733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.578375    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-secret-telemeter-client\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.578733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.578423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-metrics-client-ca\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.578733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.578457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-telemeter-client-tls\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.579006 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.578748    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-serving-certs-ca-bundle\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.579006 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.578904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.579199 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.579155    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-metrics-client-ca\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.581059 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.581037    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.581153 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.581081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2625f9b0-0053-4570-a88c-cc2172af60a5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-tdv7m\" (UID: \"2625f9b0-0053-4570-a88c-cc2172af60a5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:43.581212 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.581171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-secret-telemeter-client\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.581639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.581618    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-telemeter-client-tls\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.581723 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.581655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-federate-client-tls\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.591691 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.591663    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhr7r\" (UniqueName: \"kubernetes.io/projected/b5f6234f-a6e3-4759-9806-ff6844d5a6dd-kube-api-access-mhr7r\") pod \"telemeter-client-67fc864b6-ss9cn\" (UID: \"b5f6234f-a6e3-4759-9806-ff6844d5a6dd\") " pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.731722 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.731638    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn"
Sep 08 00:34:43.784336 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:43.784300    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:44.557750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.557711    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:34:44.561666 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.561641    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.572434 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.572399    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 00:34:44.572434 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.572400    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 00:34:44.572646 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.572505    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 00:34:44.573883 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.573857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 00:34:44.573991 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.573893    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 00:34:44.573991 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.573942    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 00:34:44.578324 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.578294    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 00:34:44.578324 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.578299    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ect9c9p30l9bg\""
Sep 08 00:34:44.578489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.578326    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 00:34:44.585406 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.585379    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 00:34:44.585512 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.585464    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 00:34:44.585774 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.585757    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 00:34:44.585774 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.585768    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sktpq\""
Sep 08 00:34:44.585928 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.585760    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 00:34:44.586282 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.586264    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 00:34:44.591563 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.591538    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 00:34:44.599447 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.599181    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 00:34:44.601740 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.601718    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:34:44.687207 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687352 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687352 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687245    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687352 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687273    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687492 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687492 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4qd\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-kube-api-access-vd4qd\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687492 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687492 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687492 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687457    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687721 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687564    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687721 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687651    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687721 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-config\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687721 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687788    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687829    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687858    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.687896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687877    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.688125 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687914    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.688125 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.687963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.788922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.788882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.788936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.788959    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-config\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.788974    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.789013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.789039    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.789063    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.789084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789434 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.789111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.789434 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.789142    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.790409 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.790084    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.790409 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.790154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792392    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792555    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792680    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4qd\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-kube-api-access-vd4qd\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792708    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792754    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.792910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.792870    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.793576 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.793162    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.793576 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.793368    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.793576 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.793443    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.794910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.793866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.794910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.794062    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.794910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.794104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.794910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.794704    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.794910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.794819    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.794910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.794858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.795834 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.795812    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.795953 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.795864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.796017 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.795991    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-config\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.796444 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.796423    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.797433 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.797413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.805526 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.805488    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4qd\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-kube-api-access-vd4qd\") pod \"prometheus-k8s-0\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:44.873577 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:44.873491    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:34:45.636124 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:45.636092    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:34:45.667881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:45.667739    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57679cb458-2t89t"]
Sep 08 00:34:47.738507 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.738421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"]
Sep 08 00:34:47.744205 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:47.744175    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2625f9b0_0053_4570_a88c_cc2172af60a5.slice/crio-af48f79ee78757cb1699d6150597d128cfd6ff572058c84824217cd089b784b6 WatchSource:0}: Error finding container af48f79ee78757cb1699d6150597d128cfd6ff572058c84824217cd089b784b6: Status 404 returned error can't find the container with id af48f79ee78757cb1699d6150597d128cfd6ff572058c84824217cd089b784b6
Sep 08 00:34:47.749048 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.748990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms" event={"ID":"a0576d64-f3ba-4128-bca9-2ac8ec0190cd","Type":"ContainerStarted","Data":"7dd5e0d708587e37d1127fa44073759c3382c747977de7141c41300b6f2328eb"}
Sep 08 00:34:47.751938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.751908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm9p9" event={"ID":"5ce59eab-6fe6-45da-b56e-5297ec1da3db","Type":"ContainerStarted","Data":"95bdf004b27f4ca0a5af80a24611bb7a05bfabb3dc69a8b1735768a04ed09697"}
Sep 08 00:34:47.752037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.751968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pm9p9" event={"ID":"5ce59eab-6fe6-45da-b56e-5297ec1da3db","Type":"ContainerStarted","Data":"01d97c2526d69b4b79ea84a552493c892baa53d783e26c9d17ebe8d6a536a500"}
Sep 08 00:34:47.753885 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.753853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-25kz4" event={"ID":"67ed3f60-57cd-4843-96e9-e81b35eae7c5","Type":"ContainerStarted","Data":"d4f3970e1fd3deb2004f7e09e107a2e18c95a65a979b71300d1d34fc5b368a79"}
Sep 08 00:34:47.754373 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.754350    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-25kz4"
Sep 08 00:34:47.755461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.755429    2555 patch_prober.go:28] interesting pod/downloads-85d98c998f-25kz4 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused" start-of-body=
Sep 08 00:34:47.755560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.755484    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-25kz4" podUID="67ed3f60-57cd-4843-96e9-e81b35eae7c5" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused"
Sep 08 00:34:47.759282 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.759023    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57679cb458-2t89t"]
Sep 08 00:34:47.759636 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:47.759580    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e2354e_21c3_450f_a060_00fb2141c3a8.slice/crio-e23b77c0d20c069ccf4dac7b7caf502f61266762a2d7ebc16044cc44898142c1 WatchSource:0}: Error finding container e23b77c0d20c069ccf4dac7b7caf502f61266762a2d7ebc16044cc44898142c1: Status 404 returned error can't find the container with id e23b77c0d20c069ccf4dac7b7caf502f61266762a2d7ebc16044cc44898142c1
Sep 08 00:34:47.760279 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:47.760201    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae294dcf_ee38_4fb4_98fe_53be942da6f8.slice/crio-f5ad88534379e582bda3eb554439777ff97ed75703bcdf2063e5cc955136048e WatchSource:0}: Error finding container f5ad88534379e582bda3eb554439777ff97ed75703bcdf2063e5cc955136048e: Status 404 returned error can't find the container with id f5ad88534379e582bda3eb554439777ff97ed75703bcdf2063e5cc955136048e
Sep 08 00:34:47.762272 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.762145    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-656dc89847-bqg56"]
Sep 08 00:34:47.764573 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.764550    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:34:47.768539 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:47.768454    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3087876b_ef90_41e7_8a12_23add06d28e9.slice/crio-eec101c8b3983b9f645c960ed5949f888509832adfd155dcf963513eba95cc89 WatchSource:0}: Error finding container eec101c8b3983b9f645c960ed5949f888509832adfd155dcf963513eba95cc89: Status 404 returned error can't find the container with id eec101c8b3983b9f645c960ed5949f888509832adfd155dcf963513eba95cc89
Sep 08 00:34:47.770430 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.770170    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67fc864b6-ss9cn"]
Sep 08 00:34:47.774899 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:47.774868    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f6234f_a6e3_4759_9806_ff6844d5a6dd.slice/crio-35fb9bba4161b54b4adf210d9419142ed5302c752c34164b6824efadd8012066 WatchSource:0}: Error finding container 35fb9bba4161b54b4adf210d9419142ed5302c752c34164b6824efadd8012066: Status 404 returned error can't find the container with id 35fb9bba4161b54b4adf210d9419142ed5302c752c34164b6824efadd8012066
Sep 08 00:34:47.805666 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.805517    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pm9p9" podStartSLOduration=8.631154443 podStartE2EDuration="9.805496206s" totalImagesPullingTime="1.103086282s" totalInitContainerRuntime="71.255481ms" isStatefulPod=true podCreationTimestamp="2026-09-08 00:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:47.804399749 +0000 UTC m=+122.147702812" watchObservedRunningTime="2026-09-08 00:34:47.805496206 +0000 UTC m=+122.148799227"
Sep 08 00:34:47.844984 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:47.844913    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-25kz4" podStartSLOduration=1.538000343 podStartE2EDuration="18.84488018s" totalImagesPullingTime="17.306879837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:47.842814031 +0000 UTC m=+122.186117051" watchObservedRunningTime="2026-09-08 00:34:47.84488018 +0000 UTC m=+122.188183200"
Sep 08 00:34:48.766555 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:48.766414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" event={"ID":"92e2354e-21c3-450f-a060-00fb2141c3a8","Type":"ContainerStarted","Data":"e23b77c0d20c069ccf4dac7b7caf502f61266762a2d7ebc16044cc44898142c1"}
Sep 08 00:34:48.774689 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:48.774609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m" event={"ID":"2625f9b0-0053-4570-a88c-cc2172af60a5","Type":"ContainerStarted","Data":"af48f79ee78757cb1699d6150597d128cfd6ff572058c84824217cd089b784b6"}
Sep 08 00:34:48.780224 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:48.780121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn" event={"ID":"b5f6234f-a6e3-4759-9806-ff6844d5a6dd","Type":"ContainerStarted","Data":"35fb9bba4161b54b4adf210d9419142ed5302c752c34164b6824efadd8012066"}
Sep 08 00:34:48.784036 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:48.783938    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57679cb458-2t89t" event={"ID":"ae294dcf-ee38-4fb4-98fe-53be942da6f8","Type":"ContainerStarted","Data":"f5ad88534379e582bda3eb554439777ff97ed75703bcdf2063e5cc955136048e"}
Sep 08 00:34:48.788487 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:48.788361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerStarted","Data":"eec101c8b3983b9f645c960ed5949f888509832adfd155dcf963513eba95cc89"}
Sep 08 00:34:48.798406 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:48.798119    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-25kz4"
Sep 08 00:34:51.502489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:51.502448    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b8f8c9778-pzvz5"]
Sep 08 00:34:54.821477 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.821296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57679cb458-2t89t" event={"ID":"ae294dcf-ee38-4fb4-98fe-53be942da6f8","Type":"ContainerStarted","Data":"b14b59b384ff8f7f1fb6b14a2b3e8c98fc6e0d0c145412ef03ec9593220c6dbc"}
Sep 08 00:34:54.824146 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.824080    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms" event={"ID":"a0576d64-f3ba-4128-bca9-2ac8ec0190cd","Type":"ContainerStarted","Data":"eeb2f151af9039e50928e31b214dce0e97c3bb96ab2238007f25662b554fccb1"}
Sep 08 00:34:54.826570 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.826545    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" event={"ID":"92e2354e-21c3-450f-a060-00fb2141c3a8","Type":"ContainerStarted","Data":"0ec472d3e9913c8d63c6d04aa49ecdc8895e402e09a5311acd92863a6c14b89d"}
Sep 08 00:34:54.826681 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.826635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" event={"ID":"92e2354e-21c3-450f-a060-00fb2141c3a8","Type":"ContainerStarted","Data":"b611f6545441090e75f72f7a1cefbb4be54a2ecad62b22cc98eae17c9b39bf07"}
Sep 08 00:34:54.828975 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.828951    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m" event={"ID":"2625f9b0-0053-4570-a88c-cc2172af60a5","Type":"ContainerStarted","Data":"c1b294b4cd1eaeeba9be811294958639e5bf9bedcd114c72ac72ef3240b79793"}
Sep 08 00:34:54.831632 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.831607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn" event={"ID":"b5f6234f-a6e3-4759-9806-ff6844d5a6dd","Type":"ContainerStarted","Data":"332e859bac1c72a2ecca9688a98427961f7d8f9d0b60e84ad9fdb790368d0303"}
Sep 08 00:34:54.833366 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.833233    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:54.838937 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.838919    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m"
Sep 08 00:34:54.867168 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.866502    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g2gms" podStartSLOduration=9.964253648 podStartE2EDuration="16.866487993s" totalImagesPullingTime="6.902234345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:54.86598859 +0000 UTC m=+129.209291628" watchObservedRunningTime="2026-09-08 00:34:54.866487993 +0000 UTC m=+129.209791042"
Sep 08 00:34:54.867168 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.866817    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57679cb458-2t89t" podStartSLOduration=8.147306526 podStartE2EDuration="14.866810433s" totalImagesPullingTime="6.719503907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:54.846397126 +0000 UTC m=+129.189700147" watchObservedRunningTime="2026-09-08 00:34:54.866810433 +0000 UTC m=+129.210113453"
Sep 08 00:34:54.882492 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:54.882449    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-tdv7m" podStartSLOduration=6.146866702 podStartE2EDuration="12.882432126s" totalImagesPullingTime="6.735565424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:54.881232886 +0000 UTC m=+129.224535906" watchObservedRunningTime="2026-09-08 00:34:54.882432126 +0000 UTC m=+129.225735146"
Sep 08 00:34:55.004547 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.004462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:34:55.007234 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.007205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd337a55-7af0-4b6c-98e9-a4980713cb79-metrics-certs\") pod \"network-metrics-daemon-gmnq8\" (UID: \"bd337a55-7af0-4b6c-98e9-a4980713cb79\") " pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:34:55.273462 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.273377    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xvx4\""
Sep 08 00:34:55.280936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.280886    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gmnq8"
Sep 08 00:34:55.577483 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.577458    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gmnq8"]
Sep 08 00:34:55.580330 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:34:55.580291    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd337a55_7af0_4b6c_98e9_a4980713cb79.slice/crio-b3dc5d1bd64ea1d1883d8940f4f52d2abd630736c1a05e0a0563f72ce04ec6c4 WatchSource:0}: Error finding container b3dc5d1bd64ea1d1883d8940f4f52d2abd630736c1a05e0a0563f72ce04ec6c4: Status 404 returned error can't find the container with id b3dc5d1bd64ea1d1883d8940f4f52d2abd630736c1a05e0a0563f72ce04ec6c4
Sep 08 00:34:55.845463 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.845423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gmnq8" event={"ID":"bd337a55-7af0-4b6c-98e9-a4980713cb79","Type":"ContainerStarted","Data":"b3dc5d1bd64ea1d1883d8940f4f52d2abd630736c1a05e0a0563f72ce04ec6c4"}
Sep 08 00:34:55.848043 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.848012    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" event={"ID":"92e2354e-21c3-450f-a060-00fb2141c3a8","Type":"ContainerStarted","Data":"97b4a8e5e60848717a11b58b27e9be323b4ab482d27d258619ef87eab1e1099f"}
Sep 08 00:34:55.850464 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.850438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn" event={"ID":"b5f6234f-a6e3-4759-9806-ff6844d5a6dd","Type":"ContainerStarted","Data":"6a777d8b9fb617d27bd8c81a9fa57b3a4f26cfba6f1803a86a092e2c679ce357"}
Sep 08 00:34:55.850566 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.850489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn" event={"ID":"b5f6234f-a6e3-4759-9806-ff6844d5a6dd","Type":"ContainerStarted","Data":"30551a478e3a87431133d6215834c2f5eba5830919183139454fd6d47217216e"}
Sep 08 00:34:55.852358 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.852329    2555 generic.go:356] "Generic (PLEG): container finished" podID="3087876b-ef90-41e7-8a12-23add06d28e9" containerID="d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7" exitCode=0
Sep 08 00:34:55.852465 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.852427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7"}
Sep 08 00:34:55.880868 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:55.880780    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67fc864b6-ss9cn" podStartSLOduration=5.279795425 podStartE2EDuration="12.880759751s" totalImagesPullingTime="7.600964326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:55.880206746 +0000 UTC m=+130.223509767" watchObservedRunningTime="2026-09-08 00:34:55.880759751 +0000 UTC m=+130.224062772"
Sep 08 00:34:56.865170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:56.865130    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" event={"ID":"92e2354e-21c3-450f-a060-00fb2141c3a8","Type":"ContainerStarted","Data":"338ded3438e8338c974a40417d37bbf7fdbe4377fe6fbb0a701ddc827ec13939"}
Sep 08 00:34:56.865642 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:56.865225    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" event={"ID":"92e2354e-21c3-450f-a060-00fb2141c3a8","Type":"ContainerStarted","Data":"f3ba7eb0913daa3ad7f90127f0b917ac914c9b0e45b0c68a0bd46c690a2f00ca"}
Sep 08 00:34:57.873003 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:57.872960    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gmnq8" event={"ID":"bd337a55-7af0-4b6c-98e9-a4980713cb79","Type":"ContainerStarted","Data":"2fb0c265cd076c26a4bee59b1e416bdea0c1d745a29450fefc6afffbd9183143"}
Sep 08 00:34:57.876668 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:57.876637    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" event={"ID":"92e2354e-21c3-450f-a060-00fb2141c3a8","Type":"ContainerStarted","Data":"4fcc8bec83697409b1f55d1b23806fd9e24b494bf74e11e537c9e2305fcae150"}
Sep 08 00:34:57.876814 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:57.876794    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:34:57.909272 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:57.909098    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56" podStartSLOduration=8.598760124 podStartE2EDuration="16.909077414s" totalImagesPullingTime="8.31031729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:57.907996701 +0000 UTC m=+132.251299803" watchObservedRunningTime="2026-09-08 00:34:57.909077414 +0000 UTC m=+132.252380433"
Sep 08 00:34:58.883668 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:58.883631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gmnq8" event={"ID":"bd337a55-7af0-4b6c-98e9-a4980713cb79","Type":"ContainerStarted","Data":"04ff59eeb0411ede9e0de0f3b6fc094524b411e6a8a7cf702d0fd9699f0acb82"}
Sep 08 00:34:58.907360 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:58.907310    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gmnq8" podStartSLOduration=130.894791211 podStartE2EDuration="2m12.907290055s" totalImagesPullingTime="2.012498844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:58.904608897 +0000 UTC m=+133.247911915" watchObservedRunningTime="2026-09-08 00:34:58.907290055 +0000 UTC m=+133.250593075"
Sep 08 00:34:59.893614 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:59.893566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerStarted","Data":"f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082"}
Sep 08 00:34:59.893995 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:59.893676    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerStarted","Data":"2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b"}
Sep 08 00:34:59.893995 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:34:59.893725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerStarted","Data":"42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b"}
Sep 08 00:35:00.874638 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:00.874603    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:35:00.900359 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:00.900327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerStarted","Data":"ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860"}
Sep 08 00:35:00.900733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:00.900384    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerStarted","Data":"d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b"}
Sep 08 00:35:00.900733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:00.900409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerStarted","Data":"e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a"}
Sep 08 00:35:00.940482 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:00.940423    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.219351669 podStartE2EDuration="16.940403155s" totalImagesPullingTime="10.693454278s" totalInitContainerRuntime="27.597208ms" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:35:00.937731232 +0000 UTC m=+135.281034253" watchObservedRunningTime="2026-09-08 00:35:00.940403155 +0000 UTC m=+135.283706175"
Sep 08 00:35:03.890259 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:03.890229    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-656dc89847-bqg56"
Sep 08 00:35:04.873906 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:04.873862    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:35:16.526101 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.526033    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5" podUID="91a90b67-5af0-46df-b5dd-c2b7620c51ff" containerName="registry" containerID="cri-o://5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90" gracePeriod=30
Sep 08 00:35:16.821009 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.820981    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:35:16.886215 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886180    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91a90b67-5af0-46df-b5dd-c2b7620c51ff-ca-trust-extracted\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.886361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886230    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-image-registry-private-configuration\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.886361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886253    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.886361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886270    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-bound-sa-token\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.886361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886294    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-installation-pull-secrets\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.886361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886332    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnfgq\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-kube-api-access-rnfgq\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.886361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886348    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-certificates\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.886733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.886369    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-trusted-ca\") pod \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\" (UID: \"91a90b67-5af0-46df-b5dd-c2b7620c51ff\") "
Sep 08 00:35:16.887097 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.887009    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:35:16.887097 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.887057    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:35:16.888999 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.888971    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:35:16.889110 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.889047    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-kube-api-access-rnfgq" (OuterVolumeSpecName: "kube-api-access-rnfgq") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "kube-api-access-rnfgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:35:16.889187 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.889166    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:35:16.889232 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.889179    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:35:16.889232 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.889188    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:35:16.894459 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.894433    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a90b67-5af0-46df-b5dd-c2b7620c51ff-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "91a90b67-5af0-46df-b5dd-c2b7620c51ff" (UID: "91a90b67-5af0-46df-b5dd-c2b7620c51ff"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:35:16.967999 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.967973    2555 generic.go:356] "Generic (PLEG): container finished" podID="91a90b67-5af0-46df-b5dd-c2b7620c51ff" containerID="5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90" exitCode=0
Sep 08 00:35:16.968113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.968019    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5"
Sep 08 00:35:16.968113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.968054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5" event={"ID":"91a90b67-5af0-46df-b5dd-c2b7620c51ff","Type":"ContainerDied","Data":"5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90"}
Sep 08 00:35:16.968113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.968107    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b8f8c9778-pzvz5" event={"ID":"91a90b67-5af0-46df-b5dd-c2b7620c51ff","Type":"ContainerDied","Data":"2f3719ee7dc6cd477f1552dcd00c16383efad7c93f0ff5f4e78c1a18e3083416"}
Sep 08 00:35:16.968292 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.968131    2555 scope.go:117] "RemoveContainer" containerID="5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90"
Sep 08 00:35:16.976786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.976767    2555 scope.go:117] "RemoveContainer" containerID="5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90"
Sep 08 00:35:16.977088 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:35:16.977066    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90\": container with ID starting with 5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90 not found: ID does not exist" containerID="5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90"
Sep 08 00:35:16.977170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.977095    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90"} err="failed to get container status \"5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90\": rpc error: code = NotFound desc = could not find container \"5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90\": container with ID starting with 5618c9aa50ce4bd345be56e13cb57bfe2e295f37ec0164ea3d27699192af2f90 not found: ID does not exist"
Sep 08 00:35:16.987925 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987902    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/91a90b67-5af0-46df-b5dd-c2b7620c51ff-ca-trust-extracted\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.987925 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987923    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-image-registry-private-configuration\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.988032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987934    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.988032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987942    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-bound-sa-token\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.988032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987950    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/91a90b67-5af0-46df-b5dd-c2b7620c51ff-installation-pull-secrets\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.988032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987958    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnfgq\" (UniqueName: \"kubernetes.io/projected/91a90b67-5af0-46df-b5dd-c2b7620c51ff-kube-api-access-rnfgq\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.988032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987966    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-registry-certificates\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.988032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.987974    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91a90b67-5af0-46df-b5dd-c2b7620c51ff-trusted-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:16.993521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.993501    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b8f8c9778-pzvz5"]
Sep 08 00:35:16.997667 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:16.997646    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b8f8c9778-pzvz5"]
Sep 08 00:35:18.155537 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:18.153310    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a90b67-5af0-46df-b5dd-c2b7620c51ff" path="/var/lib/kubelet/pods/91a90b67-5af0-46df-b5dd-c2b7620c51ff/volumes"
Sep 08 00:35:19.852461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:19.852426    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57679cb458-2t89t" podUID="ae294dcf-ee38-4fb4-98fe-53be942da6f8" containerName="console" containerID="cri-o://b14b59b384ff8f7f1fb6b14a2b3e8c98fc6e0d0c145412ef03ec9593220c6dbc" gracePeriod=15
Sep 08 00:35:19.985305 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:19.985276    2555 generic.go:356] "Generic (PLEG): container finished" podID="ae294dcf-ee38-4fb4-98fe-53be942da6f8" containerID="b14b59b384ff8f7f1fb6b14a2b3e8c98fc6e0d0c145412ef03ec9593220c6dbc" exitCode=0
Sep 08 00:35:19.985440 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:19.985328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57679cb458-2t89t" event={"ID":"ae294dcf-ee38-4fb4-98fe-53be942da6f8","Type":"ContainerDied","Data":"b14b59b384ff8f7f1fb6b14a2b3e8c98fc6e0d0c145412ef03ec9593220c6dbc"}
Sep 08 00:35:20.111839 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.111772    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:35:20.215181 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215154    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-trusted-ca-bundle\") pod \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") "
Sep 08 00:35:20.215324 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215200    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-oauth-serving-cert\") pod \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") "
Sep 08 00:35:20.215324 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215216    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-serving-cert\") pod \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") "
Sep 08 00:35:20.215324 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-oauth-config\") pod \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") "
Sep 08 00:35:20.215324 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215306    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-config\") pod \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") "
Sep 08 00:35:20.215528 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215331    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86s8j\" (UniqueName: \"kubernetes.io/projected/ae294dcf-ee38-4fb4-98fe-53be942da6f8-kube-api-access-86s8j\") pod \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") "
Sep 08 00:35:20.215528 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215354    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-service-ca\") pod \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\" (UID: \"ae294dcf-ee38-4fb4-98fe-53be942da6f8\") "
Sep 08 00:35:20.215655 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215559    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ae294dcf-ee38-4fb4-98fe-53be942da6f8" (UID: "ae294dcf-ee38-4fb4-98fe-53be942da6f8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:35:20.215841 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215781    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ae294dcf-ee38-4fb4-98fe-53be942da6f8" (UID: "ae294dcf-ee38-4fb4-98fe-53be942da6f8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:35:20.215947 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215907    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-config" (OuterVolumeSpecName: "console-config") pod "ae294dcf-ee38-4fb4-98fe-53be942da6f8" (UID: "ae294dcf-ee38-4fb4-98fe-53be942da6f8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:35:20.215947 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.215929    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-service-ca" (OuterVolumeSpecName: "service-ca") pod "ae294dcf-ee38-4fb4-98fe-53be942da6f8" (UID: "ae294dcf-ee38-4fb4-98fe-53be942da6f8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:35:20.217753 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.217726    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae294dcf-ee38-4fb4-98fe-53be942da6f8-kube-api-access-86s8j" (OuterVolumeSpecName: "kube-api-access-86s8j") pod "ae294dcf-ee38-4fb4-98fe-53be942da6f8" (UID: "ae294dcf-ee38-4fb4-98fe-53be942da6f8"). InnerVolumeSpecName "kube-api-access-86s8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:35:20.217955 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.217932    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ae294dcf-ee38-4fb4-98fe-53be942da6f8" (UID: "ae294dcf-ee38-4fb4-98fe-53be942da6f8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:35:20.218034 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.217956    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ae294dcf-ee38-4fb4-98fe-53be942da6f8" (UID: "ae294dcf-ee38-4fb4-98fe-53be942da6f8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:35:20.316534 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.316482    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-oauth-serving-cert\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:20.316534 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.316528    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-serving-cert\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:20.316534 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.316538    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-oauth-config\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:20.316534 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.316548    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-console-config\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:20.316827 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.316559    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86s8j\" (UniqueName: \"kubernetes.io/projected/ae294dcf-ee38-4fb4-98fe-53be942da6f8-kube-api-access-86s8j\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:20.316827 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.316568    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:20.316827 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.316576    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae294dcf-ee38-4fb4-98fe-53be942da6f8-trusted-ca-bundle\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:35:20.990941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.990912    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57679cb458-2t89t"
Sep 08 00:35:20.990941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.990937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57679cb458-2t89t" event={"ID":"ae294dcf-ee38-4fb4-98fe-53be942da6f8","Type":"ContainerDied","Data":"f5ad88534379e582bda3eb554439777ff97ed75703bcdf2063e5cc955136048e"}
Sep 08 00:35:20.991394 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:20.990989    2555 scope.go:117] "RemoveContainer" containerID="b14b59b384ff8f7f1fb6b14a2b3e8c98fc6e0d0c145412ef03ec9593220c6dbc"
Sep 08 00:35:21.009944 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:21.009920    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57679cb458-2t89t"]
Sep 08 00:35:21.013728 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:21.013708    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57679cb458-2t89t"]
Sep 08 00:35:22.152533 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:22.152501    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae294dcf-ee38-4fb4-98fe-53be942da6f8" path="/var/lib/kubelet/pods/ae294dcf-ee38-4fb4-98fe-53be942da6f8/volumes"
Sep 08 00:35:44.874695 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:44.874617    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:35:44.893982 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:44.893952    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:35:45.107192 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:35:45.107162    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:02.744891 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:02.744851    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:36:02.745455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:02.745163    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="prometheus" containerID="cri-o://42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b" gracePeriod=600
Sep 08 00:36:02.745455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:02.745184    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy" containerID="cri-o://d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b" gracePeriod=600
Sep 08 00:36:02.745455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:02.745219    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="thanos-sidecar" containerID="cri-o://f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082" gracePeriod=600
Sep 08 00:36:02.745455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:02.745233    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860" gracePeriod=600
Sep 08 00:36:02.745455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:02.745256    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-web" containerID="cri-o://e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a" gracePeriod=600
Sep 08 00:36:02.745455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:02.745248    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="config-reloader" containerID="cri-o://2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b" gracePeriod=600
Sep 08 00:36:03.162817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162779    2555 generic.go:356] "Generic (PLEG): container finished" podID="3087876b-ef90-41e7-8a12-23add06d28e9" containerID="ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860" exitCode=0
Sep 08 00:36:03.162817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162803    2555 generic.go:356] "Generic (PLEG): container finished" podID="3087876b-ef90-41e7-8a12-23add06d28e9" containerID="d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b" exitCode=0
Sep 08 00:36:03.162817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162809    2555 generic.go:356] "Generic (PLEG): container finished" podID="3087876b-ef90-41e7-8a12-23add06d28e9" containerID="f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082" exitCode=0
Sep 08 00:36:03.162817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162815    2555 generic.go:356] "Generic (PLEG): container finished" podID="3087876b-ef90-41e7-8a12-23add06d28e9" containerID="2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b" exitCode=0
Sep 08 00:36:03.162817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162820    2555 generic.go:356] "Generic (PLEG): container finished" podID="3087876b-ef90-41e7-8a12-23add06d28e9" containerID="42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b" exitCode=0
Sep 08 00:36:03.163098 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860"}
Sep 08 00:36:03.163098 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b"}
Sep 08 00:36:03.163098 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162916    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082"}
Sep 08 00:36:03.163098 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b"}
Sep 08 00:36:03.163098 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.162961    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b"}
Sep 08 00:36:03.994087 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:03.994052    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.170028 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.169933    2555 generic.go:356] "Generic (PLEG): container finished" podID="3087876b-ef90-41e7-8a12-23add06d28e9" containerID="e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a" exitCode=0
Sep 08 00:36:04.170028 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.170002    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a"}
Sep 08 00:36:04.170028 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.170026    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.170251 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.170072    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3087876b-ef90-41e7-8a12-23add06d28e9","Type":"ContainerDied","Data":"eec101c8b3983b9f645c960ed5949f888509832adfd155dcf963513eba95cc89"}
Sep 08 00:36:04.170251 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.170107    2555 scope.go:117] "RemoveContainer" containerID="ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860"
Sep 08 00:36:04.177801 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.177784    2555 scope.go:117] "RemoveContainer" containerID="d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b"
Sep 08 00:36:04.181989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.181958    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-metrics-client-ca\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.181989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.181991    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-trusted-ca-bundle\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182020    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-config-out\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182047    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182075    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-kubelet-serving-ca-bundle\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182096    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-kube-rbac-proxy\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-2\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182156    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-1\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182182    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd4qd\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-kube-api-access-vd4qd\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182216    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-metrics-client-certs\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182243    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-grpc-tls\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182269    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-web-config\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182312    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-0\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182341    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-tls-assets\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182379    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-config\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182403    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-thanos-prometheus-http-client-file\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182417    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:36:04.182471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182430    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-serving-certs-ca-bundle\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182505    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182646    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-db\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182686    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-tls\") pod \"3087876b-ef90-41e7-8a12-23add06d28e9\" (UID: \"3087876b-ef90-41e7-8a12-23add06d28e9\") "
Sep 08 00:36:04.182969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182735    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:36:04.182969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182783    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:36:04.182969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.182823    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:36:04.183260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.183105    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.183260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.183127    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-metrics-client-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.183260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.183143    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.183260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.183159    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.183260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.183229    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:36:04.183535 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.183511    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:36:04.185524 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.185273    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:36:04.185524 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.185398    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-config-out" (OuterVolumeSpecName: "config-out") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:36:04.186632 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.186427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.187191 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.186787    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.187191 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.186805    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.187191 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.187149    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:36:04.187191 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.187182    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:36:04.187525 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.187264    2555 scope.go:117] "RemoveContainer" containerID="e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a"
Sep 08 00:36:04.187525 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.187308    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-kube-api-access-vd4qd" (OuterVolumeSpecName: "kube-api-access-vd4qd") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "kube-api-access-vd4qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:36:04.188258 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.188205    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.188258 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.188221    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-config" (OuterVolumeSpecName: "config") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.188258 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.188240    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.188715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.188677    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.188715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.188700    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.199807 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.199777    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-web-config" (OuterVolumeSpecName: "web-config") pod "3087876b-ef90-41e7-8a12-23add06d28e9" (UID: "3087876b-ef90-41e7-8a12-23add06d28e9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:36:04.205634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.205609    2555 scope.go:117] "RemoveContainer" containerID="f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082"
Sep 08 00:36:04.213124 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.213107    2555 scope.go:117] "RemoveContainer" containerID="2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b"
Sep 08 00:36:04.220408 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.220381    2555 scope.go:117] "RemoveContainer" containerID="42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b"
Sep 08 00:36:04.227974 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.227953    2555 scope.go:117] "RemoveContainer" containerID="d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7"
Sep 08 00:36:04.235009 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.234990    2555 scope.go:117] "RemoveContainer" containerID="ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860"
Sep 08 00:36:04.235307 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:36:04.235283    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860\": container with ID starting with ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860 not found: ID does not exist" containerID="ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860"
Sep 08 00:36:04.235353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.235314    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860"} err="failed to get container status \"ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860\": rpc error: code = NotFound desc = could not find container \"ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860\": container with ID starting with ad0bad10f071f7b285fa18c58a22ca6cd55e507a5a6e64a744d653d573d49860 not found: ID does not exist"
Sep 08 00:36:04.235353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.235331    2555 scope.go:117] "RemoveContainer" containerID="d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b"
Sep 08 00:36:04.235611 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:36:04.235575    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b\": container with ID starting with d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b not found: ID does not exist" containerID="d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b"
Sep 08 00:36:04.235669 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.235610    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b"} err="failed to get container status \"d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b\": rpc error: code = NotFound desc = could not find container \"d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b\": container with ID starting with d61f54b78b8ef123075e2cb353118d99378e85683d4db10f9d3cb0ecbe6ea36b not found: ID does not exist"
Sep 08 00:36:04.235669 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.235625    2555 scope.go:117] "RemoveContainer" containerID="e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a"
Sep 08 00:36:04.235856 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:36:04.235840    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a\": container with ID starting with e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a not found: ID does not exist" containerID="e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a"
Sep 08 00:36:04.235895 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.235859    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a"} err="failed to get container status \"e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a\": rpc error: code = NotFound desc = could not find container \"e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a\": container with ID starting with e9e6bc0708071495d468a93ef71e3e64eb57d598276d7395d96edaf4067dd00a not found: ID does not exist"
Sep 08 00:36:04.235895 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.235871    2555 scope.go:117] "RemoveContainer" containerID="f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082"
Sep 08 00:36:04.236104 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:36:04.236087    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082\": container with ID starting with f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082 not found: ID does not exist" containerID="f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082"
Sep 08 00:36:04.236159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.236107    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082"} err="failed to get container status \"f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082\": rpc error: code = NotFound desc = could not find container \"f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082\": container with ID starting with f8f31f6a0880d24c0e5824b2c53520143415e93dc2a8869ae4a5e89f50371082 not found: ID does not exist"
Sep 08 00:36:04.236159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.236122    2555 scope.go:117] "RemoveContainer" containerID="2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b"
Sep 08 00:36:04.236336 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:36:04.236317    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b\": container with ID starting with 2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b not found: ID does not exist" containerID="2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b"
Sep 08 00:36:04.236377 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.236342    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b"} err="failed to get container status \"2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b\": rpc error: code = NotFound desc = could not find container \"2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b\": container with ID starting with 2d2a3c1f28057ed457d325d2b4980dc968555c33073fa26e42327be4a7309a9b not found: ID does not exist"
Sep 08 00:36:04.236377 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.236358    2555 scope.go:117] "RemoveContainer" containerID="42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b"
Sep 08 00:36:04.236562 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:36:04.236547    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b\": container with ID starting with 42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b not found: ID does not exist" containerID="42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b"
Sep 08 00:36:04.236649 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.236565    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b"} err="failed to get container status \"42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b\": rpc error: code = NotFound desc = could not find container \"42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b\": container with ID starting with 42ccd17782d406dfb175a5fbf02ee4ebb279291fedacfbf7ced6e4c3fca9d42b not found: ID does not exist"
Sep 08 00:36:04.236649 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.236577    2555 scope.go:117] "RemoveContainer" containerID="d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7"
Sep 08 00:36:04.236866 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:36:04.236849    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7\": container with ID starting with d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7 not found: ID does not exist" containerID="d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7"
Sep 08 00:36:04.236917 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.236871    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7"} err="failed to get container status \"d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7\": rpc error: code = NotFound desc = could not find container \"d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7\": container with ID starting with d21198efa3c05046db1645d8270d4b0dd6a20032ba11d5b7984be6f0c6406df7 not found: ID does not exist"
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284307    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284344    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-db\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284359    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284375    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3087876b-ef90-41e7-8a12-23add06d28e9-config-out\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284387    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284402    2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-kube-rbac-proxy\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284416    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284429    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284443    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4qd\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-kube-api-access-vd4qd\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284456    2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-metrics-client-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284468    2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-secret-grpc-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.284484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284481    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-web-config\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.285140 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.284973    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3087876b-ef90-41e7-8a12-23add06d28e9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.285140 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.285028    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3087876b-ef90-41e7-8a12-23add06d28e9-tls-assets\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.285140 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.285046    2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-config\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.285140 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.285060    2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3087876b-ef90-41e7-8a12-23add06d28e9-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:36:04.494026 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.494001    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:36:04.498493 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.498468    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:36:04.526031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526007    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:36:04.526315 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526303    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="init-config-reloader"
Sep 08 00:36:04.526356 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526317    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="init-config-reloader"
Sep 08 00:36:04.526356 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a90b67-5af0-46df-b5dd-c2b7620c51ff" containerName="registry"
Sep 08 00:36:04.526356 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526332    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a90b67-5af0-46df-b5dd-c2b7620c51ff" containerName="registry"
Sep 08 00:36:04.526356 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526341    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-thanos"
Sep 08 00:36:04.526356 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526346    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-thanos"
Sep 08 00:36:04.526356 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526356    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="thanos-sidecar"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526361    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="thanos-sidecar"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526370    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae294dcf-ee38-4fb4-98fe-53be942da6f8" containerName="console"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526376    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae294dcf-ee38-4fb4-98fe-53be942da6f8" containerName="console"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526382    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-web"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526387    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-web"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526396    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="prometheus"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526402    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="prometheus"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526409    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="config-reloader"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526414    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="config-reloader"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526420    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526425    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526470    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-web"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526478    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy-thanos"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526486    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="thanos-sidecar"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526491    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="kube-rbac-proxy"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526498    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a90b67-5af0-46df-b5dd-c2b7620c51ff" containerName="registry"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526504    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="prometheus"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526509    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae294dcf-ee38-4fb4-98fe-53be942da6f8" containerName="console"
Sep 08 00:36:04.526521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.526516    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" containerName="config-reloader"
Sep 08 00:36:04.530228 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.530212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.533213 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533191    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 00:36:04.533377 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533357    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 00:36:04.533517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533476    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 00:36:04.533517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533489    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 00:36:04.533742 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533545    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 00:36:04.533742 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533700    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 00:36:04.533823 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533788    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sktpq\""
Sep 08 00:36:04.533892 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533875    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 00:36:04.533944 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533931    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 00:36:04.534013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.533998    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ect9c9p30l9bg\""
Sep 08 00:36:04.534108 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.534094    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 00:36:04.534189 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.534174    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 00:36:04.534316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.534301    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 00:36:04.534374 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.534349    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 00:36:04.534374 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.534353    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 00:36:04.536484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.536464    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 00:36:04.538871 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.538853    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 00:36:04.547080 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.547058    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:36:04.688505 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688473    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688505 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688505    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688624    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688674    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688766    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688791    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688811    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688829    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5qz\" (UniqueName: \"kubernetes.io/projected/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-kube-api-access-qd5qz\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.688894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688891    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.689112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688911    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.689112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688937    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.689112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.688970    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.689112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.689001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.689112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.689044    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-config\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790490 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790490 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790490 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790528    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790553    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790643    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790685    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5qz\" (UniqueName: \"kubernetes.io/projected/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-kube-api-access-qd5qz\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790724    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.790782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790756    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790809    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790837    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790863    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-config\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790934    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.790991    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.791185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.791220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.791822 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.791802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.793709 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.793676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.793857 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.793832    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.793929 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.793889    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-config\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.793986 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.793971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.794754 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.794113    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.794754 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.794140    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.794754 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.794294    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.794754 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.794318    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.794754 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.794692    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.795078 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.794914    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.795149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.795124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.795193 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.795159    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.796213 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.796186    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.796447 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.796426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.796516 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.796499    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.796580 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.796562    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.802517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.802492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5qz\" (UniqueName: \"kubernetes.io/projected/fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df-kube-api-access-qd5qz\") pod \"prometheus-k8s-0\" (UID: \"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:04.840100 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:04.840080    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:36:05.120660 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:05.120631    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:36:05.121479 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:36:05.121444    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5905d5_3b4f_48f3_8cd5_32e3d3a9b9df.slice/crio-ade7fec32dfc937931d440a4f0c3e04361388567fa6f756dd3f7fd2815fc28fa WatchSource:0}: Error finding container ade7fec32dfc937931d440a4f0c3e04361388567fa6f756dd3f7fd2815fc28fa: Status 404 returned error can't find the container with id ade7fec32dfc937931d440a4f0c3e04361388567fa6f756dd3f7fd2815fc28fa
Sep 08 00:36:05.176758 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:05.176732    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerStarted","Data":"ade7fec32dfc937931d440a4f0c3e04361388567fa6f756dd3f7fd2815fc28fa"}
Sep 08 00:36:06.151729 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:06.151646    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3087876b-ef90-41e7-8a12-23add06d28e9" path="/var/lib/kubelet/pods/3087876b-ef90-41e7-8a12-23add06d28e9/volumes"
Sep 08 00:36:06.186047 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:06.185988    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df" containerID="45f500bc1bf4ce6dccfc243a8a0b3a06387ef92234ec5a21ba6b32941cf3c2a4" exitCode=0
Sep 08 00:36:06.186212 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:06.186086    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerDied","Data":"45f500bc1bf4ce6dccfc243a8a0b3a06387ef92234ec5a21ba6b32941cf3c2a4"}
Sep 08 00:36:07.193195 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:07.193155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerStarted","Data":"fd1bc526e65e275a13a97f2af7036c1f7c16cadb0aa30af1a9e8feb4d328af6e"}
Sep 08 00:36:07.193586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:07.193219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerStarted","Data":"bfd66319730021dbde7c496d86502c885510959e8616e39932e2e6976eafe318"}
Sep 08 00:36:07.193586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:07.193245    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerStarted","Data":"db502c5a607fe639eb5915b20f490b60a128c2b53621ecca8cbb1a8641de5df9"}
Sep 08 00:36:07.193586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:07.193276    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerStarted","Data":"7826d4bafab0137c737c2ca21a38b3ea6c3d1524bb60389a7deaa01573c47b80"}
Sep 08 00:36:07.193586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:07.193300    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerStarted","Data":"400ed99133470901110f005ab579c1e85a83d7edebd91faf6706ee0cf59b7543"}
Sep 08 00:36:07.193586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:07.193322    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df","Type":"ContainerStarted","Data":"8f9bf24646b12b11096064c9b7ef61f86bcf348471a5d184f42376bf9420e44a"}
Sep 08 00:36:07.224900 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:07.224841    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.2026653720000002 podStartE2EDuration="3.224822449s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.157077ms" isStatefulPod=false podCreationTimestamp="2026-09-08 00:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:36:07.223113672 +0000 UTC m=+201.566416691" watchObservedRunningTime="2026-09-08 00:36:07.224822449 +0000 UTC m=+201.568125468"
Sep 08 00:36:09.840782 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:36:09.840740    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:37:04.840557 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:04.840522    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:37:04.855968 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:04.855946    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:37:05.433191 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:05.433164    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:37:36.537987 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.537952    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"]
Sep 08 00:37:36.540219 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.540203    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.542661 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.542639    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 00:37:36.542822 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.542803    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j9ctc\""
Sep 08 00:37:36.542889 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.542872    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 00:37:36.550019 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.549999    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"]
Sep 08 00:37:36.629452 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.629426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.629561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.629476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfljm\" (UniqueName: \"kubernetes.io/projected/425a877b-4077-47d1-a187-cede6ccf76c6-kube-api-access-pfljm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.629561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.629535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.639185 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.639161    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"]
Sep 08 00:37:36.641556 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.641539    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.650479 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.650455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"]
Sep 08 00:37:36.730379 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.730379 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730380    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.730582 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.730582 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.730582 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730462    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bnkf\" (UniqueName: \"kubernetes.io/projected/75c33356-50c4-4bfc-9a97-bf5ec7348324-kube-api-access-9bnkf\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.730582 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfljm\" (UniqueName: \"kubernetes.io/projected/425a877b-4077-47d1-a187-cede6ccf76c6-kube-api-access-pfljm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.730779 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.730818 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.730789    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.742312 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.742277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfljm\" (UniqueName: \"kubernetes.io/projected/425a877b-4077-47d1-a187-cede6ccf76c6-kube-api-access-pfljm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.830912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.830887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.831099 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.830926    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.831099 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.830958    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bnkf\" (UniqueName: \"kubernetes.io/projected/75c33356-50c4-4bfc-9a97-bf5ec7348324-kube-api-access-9bnkf\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.831328 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.831307    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.831390 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.831354    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.839147 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.839116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bnkf\" (UniqueName: \"kubernetes.io/projected/75c33356-50c4-4bfc-9a97-bf5ec7348324-kube-api-access-9bnkf\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:36.849126 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.849104    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:37:36.950312 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:36.950276    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:37.116625 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:37.116471    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"]
Sep 08 00:37:37.118867 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:37:37.118838    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425a877b_4077_47d1_a187_cede6ccf76c6.slice/crio-1ce2c787951c928680262ccc81ff5e46697fde5f4b8ce105bd56ea6984219c64 WatchSource:0}: Error finding container 1ce2c787951c928680262ccc81ff5e46697fde5f4b8ce105bd56ea6984219c64: Status 404 returned error can't find the container with id 1ce2c787951c928680262ccc81ff5e46697fde5f4b8ce105bd56ea6984219c64
Sep 08 00:37:37.217862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:37.217830    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"]
Sep 08 00:37:37.220960 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:37:37.220929    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c33356_50c4_4bfc_9a97_bf5ec7348324.slice/crio-94e31a0b29a542007e81d71f28430aa17e18c1d8ee954003ecda92f57edac9a7 WatchSource:0}: Error finding container 94e31a0b29a542007e81d71f28430aa17e18c1d8ee954003ecda92f57edac9a7: Status 404 returned error can't find the container with id 94e31a0b29a542007e81d71f28430aa17e18c1d8ee954003ecda92f57edac9a7
Sep 08 00:37:37.539438 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:37.539357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9" event={"ID":"425a877b-4077-47d1-a187-cede6ccf76c6","Type":"ContainerStarted","Data":"1ce2c787951c928680262ccc81ff5e46697fde5f4b8ce105bd56ea6984219c64"}
Sep 08 00:37:37.540466 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:37.540444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7" event={"ID":"75c33356-50c4-4bfc-9a97-bf5ec7348324","Type":"ContainerStarted","Data":"94e31a0b29a542007e81d71f28430aa17e18c1d8ee954003ecda92f57edac9a7"}
Sep 08 00:37:43.572121 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:43.572081    2555 generic.go:356] "Generic (PLEG): container finished" podID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerID="e25ff15480bd149b7e12645e998cbaf32a3a5c63beaa5b4126c2ff4ea3b03c91" exitCode=0
Sep 08 00:37:43.572498 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:43.572162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7" event={"ID":"75c33356-50c4-4bfc-9a97-bf5ec7348324","Type":"ContainerDied","Data":"e25ff15480bd149b7e12645e998cbaf32a3a5c63beaa5b4126c2ff4ea3b03c91"}
Sep 08 00:37:43.573902 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:43.573880    2555 generic.go:356] "Generic (PLEG): container finished" podID="425a877b-4077-47d1-a187-cede6ccf76c6" containerID="caf12d3f78900f549e988e6f627113891b5c2654c092e49d23bcee84877de719" exitCode=0
Sep 08 00:37:43.574023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:43.573919    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9" event={"ID":"425a877b-4077-47d1-a187-cede6ccf76c6","Type":"ContainerDied","Data":"caf12d3f78900f549e988e6f627113891b5c2654c092e49d23bcee84877de719"}
Sep 08 00:37:44.580262 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:44.580227    2555 generic.go:356] "Generic (PLEG): container finished" podID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerID="e65ebd2ca55cf4ec272fbea80c5255d053cb10ebb5a30667286f79b02cfeaa1d" exitCode=0
Sep 08 00:37:44.580672 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:44.580309    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7" event={"ID":"75c33356-50c4-4bfc-9a97-bf5ec7348324","Type":"ContainerDied","Data":"e65ebd2ca55cf4ec272fbea80c5255d053cb10ebb5a30667286f79b02cfeaa1d"}
Sep 08 00:37:46.070642 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:46.070608    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:37:46.074358 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:46.073930    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:37:46.077201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:46.077177    2555 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 00:37:51.614155 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:51.614128    2555 generic.go:356] "Generic (PLEG): container finished" podID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerID="5637db024e3bff470592404fa54e6154a7f746977b6a88a4bf1884701c9045bc" exitCode=0
Sep 08 00:37:51.614463 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:51.614169    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7" event={"ID":"75c33356-50c4-4bfc-9a97-bf5ec7348324","Type":"ContainerDied","Data":"5637db024e3bff470592404fa54e6154a7f746977b6a88a4bf1884701c9045bc"}
Sep 08 00:37:52.739571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.739542    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:52.875853 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.875769    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-util\") pod \"75c33356-50c4-4bfc-9a97-bf5ec7348324\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") "
Sep 08 00:37:52.875853 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.875847    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-bundle\") pod \"75c33356-50c4-4bfc-9a97-bf5ec7348324\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") "
Sep 08 00:37:52.876029 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.875880    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bnkf\" (UniqueName: \"kubernetes.io/projected/75c33356-50c4-4bfc-9a97-bf5ec7348324-kube-api-access-9bnkf\") pod \"75c33356-50c4-4bfc-9a97-bf5ec7348324\" (UID: \"75c33356-50c4-4bfc-9a97-bf5ec7348324\") "
Sep 08 00:37:52.876318 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.876290    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-bundle" (OuterVolumeSpecName: "bundle") pod "75c33356-50c4-4bfc-9a97-bf5ec7348324" (UID: "75c33356-50c4-4bfc-9a97-bf5ec7348324"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:37:52.878112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.878090    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c33356-50c4-4bfc-9a97-bf5ec7348324-kube-api-access-9bnkf" (OuterVolumeSpecName: "kube-api-access-9bnkf") pod "75c33356-50c4-4bfc-9a97-bf5ec7348324" (UID: "75c33356-50c4-4bfc-9a97-bf5ec7348324"). InnerVolumeSpecName "kube-api-access-9bnkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:37:52.882775 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.882751    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-util" (OuterVolumeSpecName: "util") pod "75c33356-50c4-4bfc-9a97-bf5ec7348324" (UID: "75c33356-50c4-4bfc-9a97-bf5ec7348324"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:37:52.976624 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.976578    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-bundle\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:37:52.976624 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.976622    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bnkf\" (UniqueName: \"kubernetes.io/projected/75c33356-50c4-4bfc-9a97-bf5ec7348324-kube-api-access-9bnkf\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:37:52.976776 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:52.976632    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75c33356-50c4-4bfc-9a97-bf5ec7348324-util\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:37:53.622911 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:53.622874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7" event={"ID":"75c33356-50c4-4bfc-9a97-bf5ec7348324","Type":"ContainerDied","Data":"94e31a0b29a542007e81d71f28430aa17e18c1d8ee954003ecda92f57edac9a7"}
Sep 08 00:37:53.622911 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:53.622891    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wnrk7"
Sep 08 00:37:53.623104 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:53.622928    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e31a0b29a542007e81d71f28430aa17e18c1d8ee954003ecda92f57edac9a7"
Sep 08 00:37:58.002317 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002285    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"]
Sep 08 00:37:58.002741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002668    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerName="util"
Sep 08 00:37:58.002741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002683    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerName="util"
Sep 08 00:37:58.002741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002693    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerName="pull"
Sep 08 00:37:58.002741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002699    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerName="pull"
Sep 08 00:37:58.002741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002707    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerName="extract"
Sep 08 00:37:58.002741 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002714    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerName="extract"
Sep 08 00:37:58.002949 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.002766    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="75c33356-50c4-4bfc-9a97-bf5ec7348324" containerName="extract"
Sep 08 00:37:58.029197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.029173    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"]
Sep 08 00:37:58.029326 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.029264    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.032451 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.032429    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4zkls\""
Sep 08 00:37:58.032566 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.032469    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 00:37:58.032843 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.032819    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Sep 08 00:37:58.033632 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.033610    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Sep 08 00:37:58.033751 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.033627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 00:37:58.033751 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.033643    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Sep 08 00:37:58.115531 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.115507    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhc8\" (UniqueName: \"kubernetes.io/projected/5e788ca2-f7fe-4059-b0b8-1201445767a8-kube-api-access-bjhc8\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.115643 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.115555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.115643 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.115627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e788ca2-f7fe-4059-b0b8-1201445767a8-tmp\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.115730 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.115674    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.115730 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.115708    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e788ca2-f7fe-4059-b0b8-1201445767a8-manifests\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.115793 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.115757    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.216685 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.216655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e788ca2-f7fe-4059-b0b8-1201445767a8-tmp\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.216813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.216700    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.216813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.216719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e788ca2-f7fe-4059-b0b8-1201445767a8-manifests\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.216813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.216762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.216813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.216781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhc8\" (UniqueName: \"kubernetes.io/projected/5e788ca2-f7fe-4059-b0b8-1201445767a8-kube-api-access-bjhc8\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.216994 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.216815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.217143 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.217114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e788ca2-f7fe-4059-b0b8-1201445767a8-manifests\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.218908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.218887    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e788ca2-f7fe-4059-b0b8-1201445767a8-tmp\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.219171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.219149    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.219239 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.219185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.219291 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.219271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e788ca2-f7fe-4059-b0b8-1201445767a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.225581 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.225560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhc8\" (UniqueName: \"kubernetes.io/projected/5e788ca2-f7fe-4059-b0b8-1201445767a8-kube-api-access-bjhc8\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-qtk96\" (UID: \"5e788ca2-f7fe-4059-b0b8-1201445767a8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.340724 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.340698    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:37:58.655737 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.655713    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"]
Sep 08 00:37:58.722879 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:37:58.722844    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e788ca2_f7fe_4059_b0b8_1201445767a8.slice/crio-78e88d11e62dd9edf1bfc0a664d67cf93d313042fe713511b0c81429be9e7ca8 WatchSource:0}: Error finding container 78e88d11e62dd9edf1bfc0a664d67cf93d313042fe713511b0c81429be9e7ca8: Status 404 returned error can't find the container with id 78e88d11e62dd9edf1bfc0a664d67cf93d313042fe713511b0c81429be9e7ca8
Sep 08 00:37:58.724586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:58.724566    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:37:59.652881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:59.652841    2555 generic.go:356] "Generic (PLEG): container finished" podID="425a877b-4077-47d1-a187-cede6ccf76c6" containerID="9d005d94c716d23d31a390099ca3f9ed1bbc31703cb905c8a1e325a1a7b1250c" exitCode=0
Sep 08 00:37:59.653301 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:59.652956    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9" event={"ID":"425a877b-4077-47d1-a187-cede6ccf76c6","Type":"ContainerDied","Data":"9d005d94c716d23d31a390099ca3f9ed1bbc31703cb905c8a1e325a1a7b1250c"}
Sep 08 00:37:59.655210 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:37:59.655168    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96" event={"ID":"5e788ca2-f7fe-4059-b0b8-1201445767a8","Type":"ContainerStarted","Data":"78e88d11e62dd9edf1bfc0a664d67cf93d313042fe713511b0c81429be9e7ca8"}
Sep 08 00:38:00.661121 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:00.661078    2555 generic.go:356] "Generic (PLEG): container finished" podID="425a877b-4077-47d1-a187-cede6ccf76c6" containerID="932c090f3edee2b882a221145d4de60afe45911b738a0a6f5492395e75dc4b06" exitCode=0
Sep 08 00:38:00.661504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:00.661129    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9" event={"ID":"425a877b-4077-47d1-a187-cede6ccf76c6","Type":"ContainerDied","Data":"932c090f3edee2b882a221145d4de60afe45911b738a0a6f5492395e75dc4b06"}
Sep 08 00:38:01.800466 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:01.800432    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:38:01.950192 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:01.950103    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-util\") pod \"425a877b-4077-47d1-a187-cede6ccf76c6\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") "
Sep 08 00:38:01.950351 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:01.950208    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-bundle\") pod \"425a877b-4077-47d1-a187-cede6ccf76c6\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") "
Sep 08 00:38:01.950351 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:01.950278    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfljm\" (UniqueName: \"kubernetes.io/projected/425a877b-4077-47d1-a187-cede6ccf76c6-kube-api-access-pfljm\") pod \"425a877b-4077-47d1-a187-cede6ccf76c6\" (UID: \"425a877b-4077-47d1-a187-cede6ccf76c6\") "
Sep 08 00:38:01.950763 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:01.950726    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-bundle" (OuterVolumeSpecName: "bundle") pod "425a877b-4077-47d1-a187-cede6ccf76c6" (UID: "425a877b-4077-47d1-a187-cede6ccf76c6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:38:01.952348 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:01.952319    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425a877b-4077-47d1-a187-cede6ccf76c6-kube-api-access-pfljm" (OuterVolumeSpecName: "kube-api-access-pfljm") pod "425a877b-4077-47d1-a187-cede6ccf76c6" (UID: "425a877b-4077-47d1-a187-cede6ccf76c6"). InnerVolumeSpecName "kube-api-access-pfljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:38:01.954639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:01.954613    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-util" (OuterVolumeSpecName: "util") pod "425a877b-4077-47d1-a187-cede6ccf76c6" (UID: "425a877b-4077-47d1-a187-cede6ccf76c6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:38:02.051340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.051313    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-bundle\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:38:02.051340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.051338    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfljm\" (UniqueName: \"kubernetes.io/projected/425a877b-4077-47d1-a187-cede6ccf76c6-kube-api-access-pfljm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:38:02.051511 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.051350    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/425a877b-4077-47d1-a187-cede6ccf76c6-util\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:38:02.670456 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.670421    2555 generic.go:356] "Generic (PLEG): container finished" podID="5e788ca2-f7fe-4059-b0b8-1201445767a8" containerID="e33d5c931e83b6fdd65c98cb22c70a258d1c5ab34acb1db37574592fa9f28605" exitCode=0
Sep 08 00:38:02.670638 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.670518    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96" event={"ID":"5e788ca2-f7fe-4059-b0b8-1201445767a8","Type":"ContainerDied","Data":"e33d5c931e83b6fdd65c98cb22c70a258d1c5ab34acb1db37574592fa9f28605"}
Sep 08 00:38:02.672857 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.672829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9" event={"ID":"425a877b-4077-47d1-a187-cede6ccf76c6","Type":"ContainerDied","Data":"1ce2c787951c928680262ccc81ff5e46697fde5f4b8ce105bd56ea6984219c64"}
Sep 08 00:38:02.672960 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.672869    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqwrh9"
Sep 08 00:38:02.672960 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:02.672873    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ce2c787951c928680262ccc81ff5e46697fde5f4b8ce105bd56ea6984219c64"
Sep 08 00:38:03.679350 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:03.679304    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96" event={"ID":"5e788ca2-f7fe-4059-b0b8-1201445767a8","Type":"ContainerStarted","Data":"7ea8c861dd170f061ebf078a042b617e4af5c965b917d496d017861edf7ab095"}
Sep 08 00:38:03.679902 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:03.679401    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:38:03.706014 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:03.705962    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96" podStartSLOduration=2.416578065 podStartE2EDuration="6.705946079s" totalImagesPullingTime="3.859373693s" totalInitContainerRuntime="429.994321ms" isStatefulPod=false podCreationTimestamp="2026-09-08 00:37:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:03.703910915 +0000 UTC m=+318.047213934" watchObservedRunningTime="2026-09-08 00:38:03.705946079 +0000 UTC m=+318.049249097"
Sep 08 00:38:09.092797 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.092753    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"]
Sep 08 00:38:09.093357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.093246    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425a877b-4077-47d1-a187-cede6ccf76c6" containerName="extract"
Sep 08 00:38:09.093357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.093267    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="425a877b-4077-47d1-a187-cede6ccf76c6" containerName="extract"
Sep 08 00:38:09.093357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.093290    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425a877b-4077-47d1-a187-cede6ccf76c6" containerName="util"
Sep 08 00:38:09.093357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.093298    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="425a877b-4077-47d1-a187-cede6ccf76c6" containerName="util"
Sep 08 00:38:09.093357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.093313    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425a877b-4077-47d1-a187-cede6ccf76c6" containerName="pull"
Sep 08 00:38:09.093357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.093323    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="425a877b-4077-47d1-a187-cede6ccf76c6" containerName="pull"
Sep 08 00:38:09.093683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.093400    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="425a877b-4077-47d1-a187-cede6ccf76c6" containerName="extract"
Sep 08 00:38:09.096359 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.096339    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.098969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.098946    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Sep 08 00:38:09.099091 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.099038    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-v27xq\""
Sep 08 00:38:09.100054 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.100034    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Sep 08 00:38:09.111437 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.111413    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"]
Sep 08 00:38:09.209484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.209454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e588b11f-7024-4f4b-b7c9-8bc00a857057-tmp\") pod \"jobset-operator-56d489b4b8-tm7n7\" (UID: \"e588b11f-7024-4f4b-b7c9-8bc00a857057\") " pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.209639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.209504    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlk95\" (UniqueName: \"kubernetes.io/projected/e588b11f-7024-4f4b-b7c9-8bc00a857057-kube-api-access-zlk95\") pod \"jobset-operator-56d489b4b8-tm7n7\" (UID: \"e588b11f-7024-4f4b-b7c9-8bc00a857057\") " pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.310870 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.310840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e588b11f-7024-4f4b-b7c9-8bc00a857057-tmp\") pod \"jobset-operator-56d489b4b8-tm7n7\" (UID: \"e588b11f-7024-4f4b-b7c9-8bc00a857057\") " pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.311289 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.311264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e588b11f-7024-4f4b-b7c9-8bc00a857057-tmp\") pod \"jobset-operator-56d489b4b8-tm7n7\" (UID: \"e588b11f-7024-4f4b-b7c9-8bc00a857057\") " pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.311501 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.311478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlk95\" (UniqueName: \"kubernetes.io/projected/e588b11f-7024-4f4b-b7c9-8bc00a857057-kube-api-access-zlk95\") pod \"jobset-operator-56d489b4b8-tm7n7\" (UID: \"e588b11f-7024-4f4b-b7c9-8bc00a857057\") " pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.319403 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.319380    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlk95\" (UniqueName: \"kubernetes.io/projected/e588b11f-7024-4f4b-b7c9-8bc00a857057-kube-api-access-zlk95\") pod \"jobset-operator-56d489b4b8-tm7n7\" (UID: \"e588b11f-7024-4f4b-b7c9-8bc00a857057\") " pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.422145 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.422079    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"
Sep 08 00:38:09.691250 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.691182    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7"]
Sep 08 00:38:09.694454 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:09.694423    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode588b11f_7024_4f4b_b7c9_8bc00a857057.slice/crio-3f209118587b210be1160e6cdcc70f07a99b81382c0ad7b81a85ceaf46e97e50 WatchSource:0}: Error finding container 3f209118587b210be1160e6cdcc70f07a99b81382c0ad7b81a85ceaf46e97e50: Status 404 returned error can't find the container with id 3f209118587b210be1160e6cdcc70f07a99b81382c0ad7b81a85ceaf46e97e50
Sep 08 00:38:09.707964 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:09.707933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7" event={"ID":"e588b11f-7024-4f4b-b7c9-8bc00a857057","Type":"ContainerStarted","Data":"3f209118587b210be1160e6cdcc70f07a99b81382c0ad7b81a85ceaf46e97e50"}
Sep 08 00:38:12.724273 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:12.724201    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7" event={"ID":"e588b11f-7024-4f4b-b7c9-8bc00a857057","Type":"ContainerStarted","Data":"cc5a37dd98c9358f29a8cd445301339fe09e784eb93a0571ecfff2af83efe077"}
Sep 08 00:38:12.745474 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:12.745424    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-56d489b4b8-tm7n7" podStartSLOduration=1.02722397 podStartE2EDuration="3.745404386s" totalImagesPullingTime="2.718180416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:12.74348551 +0000 UTC m=+327.086788533" watchObservedRunningTime="2026-09-08 00:38:12.745404386 +0000 UTC m=+327.088707405"
Sep 08 00:38:14.686858 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:14.686829    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-qtk96"
Sep 08 00:38:25.307836 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.307801    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"]
Sep 08 00:38:25.311308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.311292    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.314030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.314004    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 08 00:38:25.314168 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.314060    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qxs8k\""
Sep 08 00:38:25.314168 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.314101    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 08 00:38:25.322875 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.322851    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"]
Sep 08 00:38:25.338230 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.338192    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrn9\" (UniqueName: \"kubernetes.io/projected/091e4352-f66a-483c-80cd-b2be910a69ab-kube-api-access-cdrn9\") pod \"cert-manager-operator-controller-manager-684f79c9b9-d4x2l\" (UID: \"091e4352-f66a-483c-80cd-b2be910a69ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.338368 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.338235    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/091e4352-f66a-483c-80cd-b2be910a69ab-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-d4x2l\" (UID: \"091e4352-f66a-483c-80cd-b2be910a69ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.439209 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.439173    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrn9\" (UniqueName: \"kubernetes.io/projected/091e4352-f66a-483c-80cd-b2be910a69ab-kube-api-access-cdrn9\") pod \"cert-manager-operator-controller-manager-684f79c9b9-d4x2l\" (UID: \"091e4352-f66a-483c-80cd-b2be910a69ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.439372 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.439219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/091e4352-f66a-483c-80cd-b2be910a69ab-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-d4x2l\" (UID: \"091e4352-f66a-483c-80cd-b2be910a69ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.439586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.439569    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/091e4352-f66a-483c-80cd-b2be910a69ab-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-d4x2l\" (UID: \"091e4352-f66a-483c-80cd-b2be910a69ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.447916 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.447888    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrn9\" (UniqueName: \"kubernetes.io/projected/091e4352-f66a-483c-80cd-b2be910a69ab-kube-api-access-cdrn9\") pod \"cert-manager-operator-controller-manager-684f79c9b9-d4x2l\" (UID: \"091e4352-f66a-483c-80cd-b2be910a69ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.620926 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.620842    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"
Sep 08 00:38:25.898526 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:25.898500    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l"]
Sep 08 00:38:25.900965 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:25.900933    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091e4352_f66a_483c_80cd_b2be910a69ab.slice/crio-6411b9b981d41864ed95d451618bea682c614617fd301568cb54e933c6c6767a WatchSource:0}: Error finding container 6411b9b981d41864ed95d451618bea682c614617fd301568cb54e933c6c6767a: Status 404 returned error can't find the container with id 6411b9b981d41864ed95d451618bea682c614617fd301568cb54e933c6c6767a
Sep 08 00:38:26.096567 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.096535    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"]
Sep 08 00:38:26.101054 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.101036    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"
Sep 08 00:38:26.107528 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.107505    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 08 00:38:26.107528 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.107524    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 00:38:26.107745 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.107528    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 00:38:26.107745 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.107664    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n52hv\""
Sep 08 00:38:26.112836 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.112815    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"]
Sep 08 00:38:26.146756 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.146731    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkhw\" (UniqueName: \"kubernetes.io/projected/efa27b1a-738f-461f-af89-1cf92383a7a6-kube-api-access-trkhw\") pod \"dns-operator-controller-manager-ffcddf5cd-5r2wd\" (UID: \"efa27b1a-738f-461f-af89-1cf92383a7a6\") " pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"
Sep 08 00:38:26.248030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.247967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkhw\" (UniqueName: \"kubernetes.io/projected/efa27b1a-738f-461f-af89-1cf92383a7a6-kube-api-access-trkhw\") pod \"dns-operator-controller-manager-ffcddf5cd-5r2wd\" (UID: \"efa27b1a-738f-461f-af89-1cf92383a7a6\") " pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"
Sep 08 00:38:26.255829 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.255803    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkhw\" (UniqueName: \"kubernetes.io/projected/efa27b1a-738f-461f-af89-1cf92383a7a6-kube-api-access-trkhw\") pod \"dns-operator-controller-manager-ffcddf5cd-5r2wd\" (UID: \"efa27b1a-738f-461f-af89-1cf92383a7a6\") " pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"
Sep 08 00:38:26.417828 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.417805    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"
Sep 08 00:38:26.703654 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.703422    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"]
Sep 08 00:38:26.706930 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:26.706891    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa27b1a_738f_461f_af89_1cf92383a7a6.slice/crio-bdf54f3037ec3165e9d1f8ead364aff30317e34f4e73ddc4350a697f465e2cd0 WatchSource:0}: Error finding container bdf54f3037ec3165e9d1f8ead364aff30317e34f4e73ddc4350a697f465e2cd0: Status 404 returned error can't find the container with id bdf54f3037ec3165e9d1f8ead364aff30317e34f4e73ddc4350a697f465e2cd0
Sep 08 00:38:26.789444 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.789396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l" event={"ID":"091e4352-f66a-483c-80cd-b2be910a69ab","Type":"ContainerStarted","Data":"6411b9b981d41864ed95d451618bea682c614617fd301568cb54e933c6c6767a"}
Sep 08 00:38:26.790908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:26.790869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd" event={"ID":"efa27b1a-738f-461f-af89-1cf92383a7a6","Type":"ContainerStarted","Data":"bdf54f3037ec3165e9d1f8ead364aff30317e34f4e73ddc4350a697f465e2cd0"}
Sep 08 00:38:28.581586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.581551    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"]
Sep 08 00:38:28.586881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.586861    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.590552 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.590530    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 00:38:28.591642 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.591613    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 00:38:28.591764 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.591646    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q8kwd\""
Sep 08 00:38:28.591764 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.591674    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 00:38:28.591764 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.591646    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 00:38:28.604817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.604793    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"]
Sep 08 00:38:28.670111 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.670074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjlt\" (UniqueName: \"kubernetes.io/projected/8a24c039-1af6-4761-9f9c-584a6773245c-kube-api-access-2sjlt\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.670286 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.670209    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a24c039-1af6-4761-9f9c-584a6773245c-tmp\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.670286 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.670250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a24c039-1af6-4761-9f9c-584a6773245c-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.771417 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.771376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a24c039-1af6-4761-9f9c-584a6773245c-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.771631 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.771456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjlt\" (UniqueName: \"kubernetes.io/projected/8a24c039-1af6-4761-9f9c-584a6773245c-kube-api-access-2sjlt\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.771631 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:38:28.771511    2555 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 08 00:38:28.771631 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:38:28.771616    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a24c039-1af6-4761-9f9c-584a6773245c-tls-certs podName:8a24c039-1af6-4761-9f9c-584a6773245c nodeName:}" failed. No retries permitted until 2026-09-08 00:38:29.271579024 +0000 UTC m=+343.614882039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8a24c039-1af6-4761-9f9c-584a6773245c-tls-certs") pod "kube-auth-proxy-58c5b795c4-6xw82" (UID: "8a24c039-1af6-4761-9f9c-584a6773245c") : secret "kube-auth-proxy-tls" not found
Sep 08 00:38:28.771800 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.771647    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a24c039-1af6-4761-9f9c-584a6773245c-tmp\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.774129 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.774100    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a24c039-1af6-4761-9f9c-584a6773245c-tmp\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:28.782208 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:28.782185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjlt\" (UniqueName: \"kubernetes.io/projected/8a24c039-1af6-4761-9f9c-584a6773245c-kube-api-access-2sjlt\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:29.277578 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:29.277533    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a24c039-1af6-4761-9f9c-584a6773245c-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:29.280384 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:29.280355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a24c039-1af6-4761-9f9c-584a6773245c-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-6xw82\" (UID: \"8a24c039-1af6-4761-9f9c-584a6773245c\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:29.498774 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:29.498739    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"
Sep 08 00:38:30.452826 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.452796    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82"]
Sep 08 00:38:30.551712 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.551667    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-84d9bfc694-8kk6x"]
Sep 08 00:38:30.555826 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.555809    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x"
Sep 08 00:38:30.558361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.558341    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-r5g22\""
Sep 08 00:38:30.565898 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.565870    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-84d9bfc694-8kk6x"]
Sep 08 00:38:30.589932 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.589908    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtp8\" (UniqueName: \"kubernetes.io/projected/516d511a-330b-45d4-948c-350aff8791aa-kube-api-access-hjtp8\") pod \"authorino-operator-84d9bfc694-8kk6x\" (UID: \"516d511a-330b-45d4-948c-350aff8791aa\") " pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x"
Sep 08 00:38:30.690984 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.690948    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtp8\" (UniqueName: \"kubernetes.io/projected/516d511a-330b-45d4-948c-350aff8791aa-kube-api-access-hjtp8\") pod \"authorino-operator-84d9bfc694-8kk6x\" (UID: \"516d511a-330b-45d4-948c-350aff8791aa\") " pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x"
Sep 08 00:38:30.699407 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.699376    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtp8\" (UniqueName: \"kubernetes.io/projected/516d511a-330b-45d4-948c-350aff8791aa-kube-api-access-hjtp8\") pod \"authorino-operator-84d9bfc694-8kk6x\" (UID: \"516d511a-330b-45d4-948c-350aff8791aa\") " pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x"
Sep 08 00:38:30.761630 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.761548    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"]
Sep 08 00:38:30.764997 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.764980    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.767542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.767516    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 00:38:30.767688 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.767516    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dhth8\""
Sep 08 00:38:30.767688 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.767558    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 00:38:30.777394 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.777370    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"]
Sep 08 00:38:30.814639 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.814585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd" event={"ID":"efa27b1a-738f-461f-af89-1cf92383a7a6","Type":"ContainerStarted","Data":"fa7a7e63f2e55eda89e5b5084b569293c174d51e1fa9c6d429f3845759ee6747"}
Sep 08 00:38:30.814793 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.814723    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"
Sep 08 00:38:30.817204 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.817170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l" event={"ID":"091e4352-f66a-483c-80cd-b2be910a69ab","Type":"ContainerStarted","Data":"e966a6a572b9a5b21dfc05c3c97d74e04407bc9fcd51112b8672391d49b11dd3"}
Sep 08 00:38:30.818977 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.818950    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82" event={"ID":"8a24c039-1af6-4761-9f9c-584a6773245c","Type":"ContainerStarted","Data":"77f4e98ad22204d7922c7451616054ce20f52d97bd7c7c5c2cbf12f69954c7dc"}
Sep 08 00:38:30.835834 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.835785    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd" podStartSLOduration=1.29201439 podStartE2EDuration="4.835766537s" totalImagesPullingTime="3.543752147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:30.833805424 +0000 UTC m=+345.177108444" watchObservedRunningTime="2026-09-08 00:38:30.835766537 +0000 UTC m=+345.179069556"
Sep 08 00:38:30.856344 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.856230    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-d4x2l" podStartSLOduration=1.6095606390000001 podStartE2EDuration="5.856213065s" totalImagesPullingTime="4.246652426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:30.854353674 +0000 UTC m=+345.197656694" watchObservedRunningTime="2026-09-08 00:38:30.856213065 +0000 UTC m=+345.199516086"
Sep 08 00:38:30.866890 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.866867    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x"
Sep 08 00:38:30.893349 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.893325    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5q59\" (UniqueName: \"kubernetes.io/projected/e387a42b-e277-4911-a74e-65cad7b7a90c-kube-api-access-t5q59\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.893509 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.893386    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e387a42b-e277-4911-a74e-65cad7b7a90c-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.893509 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.893485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e387a42b-e277-4911-a74e-65cad7b7a90c-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.994525 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.994349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5q59\" (UniqueName: \"kubernetes.io/projected/e387a42b-e277-4911-a74e-65cad7b7a90c-kube-api-access-t5q59\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.994525 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.994423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e387a42b-e277-4911-a74e-65cad7b7a90c-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.995155 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.995128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e387a42b-e277-4911-a74e-65cad7b7a90c-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.995793 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.995743    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e387a42b-e277-4911-a74e-65cad7b7a90c-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:30.998545 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:30.998513    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e387a42b-e277-4911-a74e-65cad7b7a90c-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:31.007206 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:31.007182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5q59\" (UniqueName: \"kubernetes.io/projected/e387a42b-e277-4911-a74e-65cad7b7a90c-kube-api-access-t5q59\") pod \"kuadrant-console-plugin-598f86468b-hdsmq\" (UID: \"e387a42b-e277-4911-a74e-65cad7b7a90c\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:31.074417 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:31.074379    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"
Sep 08 00:38:31.171585 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:31.171548    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-84d9bfc694-8kk6x"]
Sep 08 00:38:31.176083 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:31.176058    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516d511a_330b_45d4_948c_350aff8791aa.slice/crio-d346f9545d754381f8963aeb7c99b6104b0ce398f4f4cabceeda8d96c036d0cd WatchSource:0}: Error finding container d346f9545d754381f8963aeb7c99b6104b0ce398f4f4cabceeda8d96c036d0cd: Status 404 returned error can't find the container with id d346f9545d754381f8963aeb7c99b6104b0ce398f4f4cabceeda8d96c036d0cd
Sep 08 00:38:31.372903 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:31.372876    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq"]
Sep 08 00:38:31.375184 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:31.375153    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode387a42b_e277_4911_a74e_65cad7b7a90c.slice/crio-66c19fd1a656c09506ef36509746290e99ce303dba5ea35d4ebe28976319f2fd WatchSource:0}: Error finding container 66c19fd1a656c09506ef36509746290e99ce303dba5ea35d4ebe28976319f2fd: Status 404 returned error can't find the container with id 66c19fd1a656c09506ef36509746290e99ce303dba5ea35d4ebe28976319f2fd
Sep 08 00:38:31.825811 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:31.825766    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq" event={"ID":"e387a42b-e277-4911-a74e-65cad7b7a90c","Type":"ContainerStarted","Data":"66c19fd1a656c09506ef36509746290e99ce303dba5ea35d4ebe28976319f2fd"}
Sep 08 00:38:31.827404 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:31.827368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x" event={"ID":"516d511a-330b-45d4-948c-350aff8791aa","Type":"ContainerStarted","Data":"d346f9545d754381f8963aeb7c99b6104b0ce398f4f4cabceeda8d96c036d0cd"}
Sep 08 00:38:32.834860 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:32.834823    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82" event={"ID":"8a24c039-1af6-4761-9f9c-584a6773245c","Type":"ContainerStarted","Data":"e6226199efb6b169e3f34e31480b497a7bcefef0a744de8c9a29ef17a4c542aa"}
Sep 08 00:38:32.855333 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:32.855217    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-6xw82" podStartSLOduration=2.639657155 podStartE2EDuration="4.855194283s" totalImagesPullingTime="2.215537128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:32.854922208 +0000 UTC m=+347.198225245" watchObservedRunningTime="2026-09-08 00:38:32.855194283 +0000 UTC m=+347.198497305"
Sep 08 00:38:33.844481 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:33.844440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x" event={"ID":"516d511a-330b-45d4-948c-350aff8791aa","Type":"ContainerStarted","Data":"8e826343e41a5bb3721de39b49351005c60190abfeafb325cc9ca3df32d0beb8"}
Sep 08 00:38:33.873610 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:33.873545    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x" podStartSLOduration=1.6344091029999999 podStartE2EDuration="3.873526294s" totalImagesPullingTime="2.239117191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:33.871198122 +0000 UTC m=+348.214501142" watchObservedRunningTime="2026-09-08 00:38:33.873526294 +0000 UTC m=+348.216829317"
Sep 08 00:38:34.154217 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.154141    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"]
Sep 08 00:38:34.157760 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.157736    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"
Sep 08 00:38:34.161473 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.161447    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-77zqn\""
Sep 08 00:38:34.167385 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.167352    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"]
Sep 08 00:38:34.233914 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.233885    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxnx\" (UniqueName: \"kubernetes.io/projected/dd1a96b4-1280-4ad0-9f61-92719e3c2c37-kube-api-access-bxxnx\") pod \"limitador-operator-controller-manager-565895c755-zlh6m\" (UID: \"dd1a96b4-1280-4ad0-9f61-92719e3c2c37\") " pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"
Sep 08 00:38:34.334908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.334869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxnx\" (UniqueName: \"kubernetes.io/projected/dd1a96b4-1280-4ad0-9f61-92719e3c2c37-kube-api-access-bxxnx\") pod \"limitador-operator-controller-manager-565895c755-zlh6m\" (UID: \"dd1a96b4-1280-4ad0-9f61-92719e3c2c37\") " pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"
Sep 08 00:38:34.344564 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.344521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxnx\" (UniqueName: \"kubernetes.io/projected/dd1a96b4-1280-4ad0-9f61-92719e3c2c37-kube-api-access-bxxnx\") pod \"limitador-operator-controller-manager-565895c755-zlh6m\" (UID: \"dd1a96b4-1280-4ad0-9f61-92719e3c2c37\") " pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"
Sep 08 00:38:34.470976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.470885    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"
Sep 08 00:38:34.756881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.756855    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"]
Sep 08 00:38:34.758799 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:34.758770    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1a96b4_1280_4ad0_9f61_92719e3c2c37.slice/crio-40c79265ed2f513831ffe8574c596181c55d32839be165dedb7a728e866e2ced WatchSource:0}: Error finding container 40c79265ed2f513831ffe8574c596181c55d32839be165dedb7a728e866e2ced: Status 404 returned error can't find the container with id 40c79265ed2f513831ffe8574c596181c55d32839be165dedb7a728e866e2ced
Sep 08 00:38:34.851225 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.851192    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m" event={"ID":"dd1a96b4-1280-4ad0-9f61-92719e3c2c37","Type":"ContainerStarted","Data":"40c79265ed2f513831ffe8574c596181c55d32839be165dedb7a728e866e2ced"}
Sep 08 00:38:34.851573 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:34.851347    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x"
Sep 08 00:38:37.871856 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:37.871794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq" event={"ID":"e387a42b-e277-4911-a74e-65cad7b7a90c","Type":"ContainerStarted","Data":"1bb973f60d6be6678eb0cf76b9b6d1c93c72ce8bc1ed80fa353eb145d6bd995e"}
Sep 08 00:38:37.887895 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:37.887845    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-hdsmq" podStartSLOduration=1.785278896 podStartE2EDuration="7.887825076s" totalImagesPullingTime="6.10254618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:37.887375148 +0000 UTC m=+352.230678168" watchObservedRunningTime="2026-09-08 00:38:37.887825076 +0000 UTC m=+352.231128107"
Sep 08 00:38:38.878289 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:38.878247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m" event={"ID":"dd1a96b4-1280-4ad0-9f61-92719e3c2c37","Type":"ContainerStarted","Data":"6d963e113cc77629feabd9eaba35f37bc4ff76e6781dcab82ca4c158c3bf05dc"}
Sep 08 00:38:38.878746 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:38.878338    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"
Sep 08 00:38:38.900420 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:38.900367    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m" podStartSLOduration=1.692582279 podStartE2EDuration="4.900353001s" totalImagesPullingTime="3.207770722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:38.898096509 +0000 UTC m=+353.241399529" watchObservedRunningTime="2026-09-08 00:38:38.900353001 +0000 UTC m=+353.243656020"
Sep 08 00:38:41.830190 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:41.830154    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ffcddf5cd-5r2wd"
Sep 08 00:38:43.359068 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.359030    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7848477867-9xrln"]
Sep 08 00:38:43.362629 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.362611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.365073 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.365043    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 00:38:43.365304 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.365281    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 00:38:43.366256 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.366238    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 00:38:43.366312 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.366254    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddxg8\""
Sep 08 00:38:43.366360 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.366314    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 00:38:43.366403 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.366362    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 00:38:43.380113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.380086    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7848477867-9xrln"]
Sep 08 00:38:43.517459 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.517414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-webhook-cert\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.517658 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.517475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-metrics-certs\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.517658 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.517546    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7955\" (UniqueName: \"kubernetes.io/projected/fa38fdff-aea1-46ee-b0ea-af946ad7927d-kube-api-access-s7955\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.545156 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.545125    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-6dnk6"]
Sep 08 00:38:43.548501 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.548481    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.558619 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.558574    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-6dnk6"]
Sep 08 00:38:43.618750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.618674    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-webhook-cert\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.618750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.618721    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-metrics-certs\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.618943 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.618837    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7955\" (UniqueName: \"kubernetes.io/projected/fa38fdff-aea1-46ee-b0ea-af946ad7927d-kube-api-access-s7955\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.621078 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.621054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-webhook-cert\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.621210 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.621132    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-metrics-certs\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.627383 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.627362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7955\" (UniqueName: \"kubernetes.io/projected/fa38fdff-aea1-46ee-b0ea-af946ad7927d-kube-api-access-s7955\") pod \"maas-controller-7848477867-9xrln\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") " pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.672364 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.672335    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:43.720109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.719870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.720109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.719961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvl4\" (UniqueName: \"kubernetes.io/projected/e483cda4-02a1-4cdd-a652-0561e1ee1175-kube-api-access-vtvl4\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.720109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.720017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.821085 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.821047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.821255 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.821133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvl4\" (UniqueName: \"kubernetes.io/projected/e483cda4-02a1-4cdd-a652-0561e1ee1175-kube-api-access-vtvl4\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.821255 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.821181    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.823606 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.823570    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.823734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.823574    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.829919 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.829896    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvl4\" (UniqueName: \"kubernetes.io/projected/e483cda4-02a1-4cdd-a652-0561e1ee1175-kube-api-access-vtvl4\") pod \"maas-controller-5dbbcf5b8c-6dnk6\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") " pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.858794 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.858767    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:43.941625 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:43.941582    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7848477867-9xrln"]
Sep 08 00:38:43.943681 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:43.943645    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa38fdff_aea1_46ee_b0ea_af946ad7927d.slice/crio-f6a1b49eab0f23d3a1e9b92b9a42db9e533411e3cfd69d76ae4588ead1bf7458 WatchSource:0}: Error finding container f6a1b49eab0f23d3a1e9b92b9a42db9e533411e3cfd69d76ae4588ead1bf7458: Status 404 returned error can't find the container with id f6a1b49eab0f23d3a1e9b92b9a42db9e533411e3cfd69d76ae4588ead1bf7458
Sep 08 00:38:44.134938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:44.134859    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-6dnk6"]
Sep 08 00:38:44.139986 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:38:44.139959    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode483cda4_02a1_4cdd_a652_0561e1ee1175.slice/crio-1d1a62b33dde05e5388342f7f7fec21064b3f21fd3569e4a52fa5753c2700334 WatchSource:0}: Error finding container 1d1a62b33dde05e5388342f7f7fec21064b3f21fd3569e4a52fa5753c2700334: Status 404 returned error can't find the container with id 1d1a62b33dde05e5388342f7f7fec21064b3f21fd3569e4a52fa5753c2700334
Sep 08 00:38:44.913512 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:44.913465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6" event={"ID":"e483cda4-02a1-4cdd-a652-0561e1ee1175","Type":"ContainerStarted","Data":"1d1a62b33dde05e5388342f7f7fec21064b3f21fd3569e4a52fa5753c2700334"}
Sep 08 00:38:44.915221 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:44.915153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7848477867-9xrln" event={"ID":"fa38fdff-aea1-46ee-b0ea-af946ad7927d","Type":"ContainerStarted","Data":"f6a1b49eab0f23d3a1e9b92b9a42db9e533411e3cfd69d76ae4588ead1bf7458"}
Sep 08 00:38:45.860773 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:45.860739    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-84d9bfc694-8kk6x"
Sep 08 00:38:45.922732 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:45.922610    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6" event={"ID":"e483cda4-02a1-4cdd-a652-0561e1ee1175","Type":"ContainerStarted","Data":"76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b"}
Sep 08 00:38:45.922732 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:45.922692    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:45.924567 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:45.924527    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7848477867-9xrln" event={"ID":"fa38fdff-aea1-46ee-b0ea-af946ad7927d","Type":"ContainerStarted","Data":"8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f"}
Sep 08 00:38:45.924739 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:45.924643    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:45.942235 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:45.942187    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6" podStartSLOduration=1.3177073080000001 podStartE2EDuration="2.942170587s" totalImagesPullingTime="1.624463279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:45.941775181 +0000 UTC m=+360.285078202" watchObservedRunningTime="2026-09-08 00:38:45.942170587 +0000 UTC m=+360.285473635"
Sep 08 00:38:45.960766 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:45.960709    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7848477867-9xrln" podStartSLOduration=1.142861586 podStartE2EDuration="2.96069433s" totalImagesPullingTime="1.817832744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:45.958895878 +0000 UTC m=+360.302198898" watchObservedRunningTime="2026-09-08 00:38:45.96069433 +0000 UTC m=+360.303997348"
Sep 08 00:38:49.885653 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:49.885621    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-565895c755-zlh6m"
Sep 08 00:38:56.931348 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:56.931296    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:56.931836 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:56.931386    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:38:56.997955 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:56.997921    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7848477867-9xrln"]
Sep 08 00:38:56.998166 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:56.998133    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7848477867-9xrln" podUID="fa38fdff-aea1-46ee-b0ea-af946ad7927d" containerName="manager" containerID="cri-o://8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f" gracePeriod=30
Sep 08 00:38:57.238680 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.238654    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:57.339112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.339083    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7955\" (UniqueName: \"kubernetes.io/projected/fa38fdff-aea1-46ee-b0ea-af946ad7927d-kube-api-access-s7955\") pod \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") "
Sep 08 00:38:57.339247 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.339126    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-metrics-certs\") pod \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") "
Sep 08 00:38:57.339247 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.339170    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-webhook-cert\") pod \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\" (UID: \"fa38fdff-aea1-46ee-b0ea-af946ad7927d\") "
Sep 08 00:38:57.341273 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.341242    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa38fdff-aea1-46ee-b0ea-af946ad7927d-kube-api-access-s7955" (OuterVolumeSpecName: "kube-api-access-s7955") pod "fa38fdff-aea1-46ee-b0ea-af946ad7927d" (UID: "fa38fdff-aea1-46ee-b0ea-af946ad7927d"). InnerVolumeSpecName "kube-api-access-s7955". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:38:57.341273 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.341258    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fa38fdff-aea1-46ee-b0ea-af946ad7927d" (UID: "fa38fdff-aea1-46ee-b0ea-af946ad7927d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:38:57.341426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.341319    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fa38fdff-aea1-46ee-b0ea-af946ad7927d" (UID: "fa38fdff-aea1-46ee-b0ea-af946ad7927d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:38:57.440159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.440133    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-webhook-cert\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:38:57.440159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.440155    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7955\" (UniqueName: \"kubernetes.io/projected/fa38fdff-aea1-46ee-b0ea-af946ad7927d-kube-api-access-s7955\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:38:57.440309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.440167    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa38fdff-aea1-46ee-b0ea-af946ad7927d-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:38:57.988354 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.988318    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa38fdff-aea1-46ee-b0ea-af946ad7927d" containerID="8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f" exitCode=0
Sep 08 00:38:57.988750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.988365    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7848477867-9xrln"
Sep 08 00:38:57.988750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.988401    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7848477867-9xrln" event={"ID":"fa38fdff-aea1-46ee-b0ea-af946ad7927d","Type":"ContainerDied","Data":"8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f"}
Sep 08 00:38:57.988750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.988453    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7848477867-9xrln" event={"ID":"fa38fdff-aea1-46ee-b0ea-af946ad7927d","Type":"ContainerDied","Data":"f6a1b49eab0f23d3a1e9b92b9a42db9e533411e3cfd69d76ae4588ead1bf7458"}
Sep 08 00:38:57.988750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.988474    2555 scope.go:117] "RemoveContainer" containerID="8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f"
Sep 08 00:38:57.997280 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.997262    2555 scope.go:117] "RemoveContainer" containerID="8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f"
Sep 08 00:38:57.997544 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:38:57.997525    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f\": container with ID starting with 8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f not found: ID does not exist" containerID="8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f"
Sep 08 00:38:57.997630 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:57.997549    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f"} err="failed to get container status \"8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f\": rpc error: code = NotFound desc = could not find container \"8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f\": container with ID starting with 8e47e61acee8206a63fdd29d6bcf4f76ffc62ba0769f2036e95104f901f1243f not found: ID does not exist"
Sep 08 00:38:58.010346 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:58.010319    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7848477867-9xrln"]
Sep 08 00:38:58.013891 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:58.013871    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7848477867-9xrln"]
Sep 08 00:38:58.159175 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:38:58.159130    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa38fdff-aea1-46ee-b0ea-af946ad7927d" path="/var/lib/kubelet/pods/fa38fdff-aea1-46ee-b0ea-af946ad7927d/volumes"
Sep 08 00:39:13.139901 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.139862    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:39:13.140370 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.140248    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa38fdff-aea1-46ee-b0ea-af946ad7927d" containerName="manager"
Sep 08 00:39:13.140370 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.140262    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa38fdff-aea1-46ee-b0ea-af946ad7927d" containerName="manager"
Sep 08 00:39:13.140370 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.140320    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa38fdff-aea1-46ee-b0ea-af946ad7927d" containerName="manager"
Sep 08 00:39:13.143490 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.143470    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.145800 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.145778    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 00:39:13.151369 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.151346    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:39:13.178215 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.178185    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:39:13.276414 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.276380    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1374a2e4-e8ff-40e7-b603-c9b5d895e079-config-file\") pod \"limitador-limitador-85d9db4d5d-mpvgw\" (UID: \"1374a2e4-e8ff-40e7-b603-c9b5d895e079\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.276414 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.276418    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcbl\" (UniqueName: \"kubernetes.io/projected/1374a2e4-e8ff-40e7-b603-c9b5d895e079-kube-api-access-6qcbl\") pod \"limitador-limitador-85d9db4d5d-mpvgw\" (UID: \"1374a2e4-e8ff-40e7-b603-c9b5d895e079\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.377241 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.377201    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1374a2e4-e8ff-40e7-b603-c9b5d895e079-config-file\") pod \"limitador-limitador-85d9db4d5d-mpvgw\" (UID: \"1374a2e4-e8ff-40e7-b603-c9b5d895e079\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.377241 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.377245    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcbl\" (UniqueName: \"kubernetes.io/projected/1374a2e4-e8ff-40e7-b603-c9b5d895e079-kube-api-access-6qcbl\") pod \"limitador-limitador-85d9db4d5d-mpvgw\" (UID: \"1374a2e4-e8ff-40e7-b603-c9b5d895e079\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.377874 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.377854    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1374a2e4-e8ff-40e7-b603-c9b5d895e079-config-file\") pod \"limitador-limitador-85d9db4d5d-mpvgw\" (UID: \"1374a2e4-e8ff-40e7-b603-c9b5d895e079\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.385086 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.385059    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcbl\" (UniqueName: \"kubernetes.io/projected/1374a2e4-e8ff-40e7-b603-c9b5d895e079-kube-api-access-6qcbl\") pod \"limitador-limitador-85d9db4d5d-mpvgw\" (UID: \"1374a2e4-e8ff-40e7-b603-c9b5d895e079\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.454773 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.454686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:13.723125 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.723100    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:39:13.727655 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:39:13.727616    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1374a2e4_e8ff_40e7_b603_c9b5d895e079.slice/crio-fab23d3946e85d792e4a7d987cd08aa4ba0d4989dc7148d2cfaa5d38cfb9bb90 WatchSource:0}: Error finding container fab23d3946e85d792e4a7d987cd08aa4ba0d4989dc7148d2cfaa5d38cfb9bb90: Status 404 returned error can't find the container with id fab23d3946e85d792e4a7d987cd08aa4ba0d4989dc7148d2cfaa5d38cfb9bb90
Sep 08 00:39:13.811845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.811806    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fwsn6"]
Sep 08 00:39:13.815089 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.815063    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fwsn6"
Sep 08 00:39:13.818113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.818081    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kwxr7\""
Sep 08 00:39:13.823129 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.823101    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fwsn6"]
Sep 08 00:39:13.881754 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.881716    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngtc\" (UniqueName: \"kubernetes.io/projected/353d3cb0-e64d-4ee3-b35c-1354849c5598-kube-api-access-8ngtc\") pod \"authorino-c548b5d7d-fwsn6\" (UID: \"353d3cb0-e64d-4ee3-b35c-1354849c5598\") " pod="kuadrant-system/authorino-c548b5d7d-fwsn6"
Sep 08 00:39:13.982748 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.982643    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngtc\" (UniqueName: \"kubernetes.io/projected/353d3cb0-e64d-4ee3-b35c-1354849c5598-kube-api-access-8ngtc\") pod \"authorino-c548b5d7d-fwsn6\" (UID: \"353d3cb0-e64d-4ee3-b35c-1354849c5598\") " pod="kuadrant-system/authorino-c548b5d7d-fwsn6"
Sep 08 00:39:13.991561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:13.991532    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngtc\" (UniqueName: \"kubernetes.io/projected/353d3cb0-e64d-4ee3-b35c-1354849c5598-kube-api-access-8ngtc\") pod \"authorino-c548b5d7d-fwsn6\" (UID: \"353d3cb0-e64d-4ee3-b35c-1354849c5598\") " pod="kuadrant-system/authorino-c548b5d7d-fwsn6"
Sep 08 00:39:14.068880 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:14.068838    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw" event={"ID":"1374a2e4-e8ff-40e7-b603-c9b5d895e079","Type":"ContainerStarted","Data":"fab23d3946e85d792e4a7d987cd08aa4ba0d4989dc7148d2cfaa5d38cfb9bb90"}
Sep 08 00:39:14.126613 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:14.126542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fwsn6"
Sep 08 00:39:14.392959 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:14.392913    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fwsn6"]
Sep 08 00:39:14.395227 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:39:14.395197    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod353d3cb0_e64d_4ee3_b35c_1354849c5598.slice/crio-c05a014cd2b938e9d90c27410509209ff721e01ad64846df10e23798575914ad WatchSource:0}: Error finding container c05a014cd2b938e9d90c27410509209ff721e01ad64846df10e23798575914ad: Status 404 returned error can't find the container with id c05a014cd2b938e9d90c27410509209ff721e01ad64846df10e23798575914ad
Sep 08 00:39:15.079367 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:15.079324    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fwsn6" event={"ID":"353d3cb0-e64d-4ee3-b35c-1354849c5598","Type":"ContainerStarted","Data":"c05a014cd2b938e9d90c27410509209ff721e01ad64846df10e23798575914ad"}
Sep 08 00:39:17.091688 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:17.091639    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fwsn6" event={"ID":"353d3cb0-e64d-4ee3-b35c-1354849c5598","Type":"ContainerStarted","Data":"4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c"}
Sep 08 00:39:17.093249 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:17.093219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw" event={"ID":"1374a2e4-e8ff-40e7-b603-c9b5d895e079","Type":"ContainerStarted","Data":"03b21c7043425f3db3c3535a0f9ce53f4eac4738bc40aeff4e795493bc58936f"}
Sep 08 00:39:17.093377 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:17.093360    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:17.112506 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:17.112454    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-fwsn6" podStartSLOduration=1.942882676 podStartE2EDuration="4.11243815s" totalImagesPullingTime="2.169555474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:17.111152909 +0000 UTC m=+391.454455931" watchObservedRunningTime="2026-09-08 00:39:17.11243815 +0000 UTC m=+391.455741168"
Sep 08 00:39:17.134240 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:17.134178    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw" podStartSLOduration=1.300605757 podStartE2EDuration="4.134155533s" totalImagesPullingTime="2.833549776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:17.129755729 +0000 UTC m=+391.473058768" watchObservedRunningTime="2026-09-08 00:39:17.134155533 +0000 UTC m=+391.477458554"
Sep 08 00:39:20.722992 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.722952    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:39:20.759542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.759504    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"]
Sep 08 00:39:20.765671 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.765635    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:20.768659 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.768630    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n98qv\""
Sep 08 00:39:20.768924 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.768904    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 00:39:20.796234 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.796200    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"]
Sep 08 00:39:20.847365 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.847326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e916f6-825c-4858-9882-d7670a697d5b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tw9lv\" (UID: \"99e916f6-825c-4858-9882-d7670a697d5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:20.847517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.847404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hpd\" (UniqueName: \"kubernetes.io/projected/99e916f6-825c-4858-9882-d7670a697d5b-kube-api-access-t2hpd\") pod \"payload-pre-processing-59fbc47bf8-tw9lv\" (UID: \"99e916f6-825c-4858-9882-d7670a697d5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:20.948689 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.948651    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e916f6-825c-4858-9882-d7670a697d5b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tw9lv\" (UID: \"99e916f6-825c-4858-9882-d7670a697d5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:20.948864 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.948723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hpd\" (UniqueName: \"kubernetes.io/projected/99e916f6-825c-4858-9882-d7670a697d5b-kube-api-access-t2hpd\") pod \"payload-pre-processing-59fbc47bf8-tw9lv\" (UID: \"99e916f6-825c-4858-9882-d7670a697d5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:20.949460 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.949431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e916f6-825c-4858-9882-d7670a697d5b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-tw9lv\" (UID: \"99e916f6-825c-4858-9882-d7670a697d5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:20.961244 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:20.961218    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hpd\" (UniqueName: \"kubernetes.io/projected/99e916f6-825c-4858-9882-d7670a697d5b-kube-api-access-t2hpd\") pod \"payload-pre-processing-59fbc47bf8-tw9lv\" (UID: \"99e916f6-825c-4858-9882-d7670a697d5b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:21.078916 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:21.078868    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:21.394242 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:21.394211    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"]
Sep 08 00:39:21.395224 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:39:21.395190    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e916f6_825c_4858_9882_d7670a697d5b.slice/crio-441491bd2bf7e087745ce4d337f29ca3cb5992f3dceea36c6b35ca74e68113e2 WatchSource:0}: Error finding container 441491bd2bf7e087745ce4d337f29ca3cb5992f3dceea36c6b35ca74e68113e2: Status 404 returned error can't find the container with id 441491bd2bf7e087745ce4d337f29ca3cb5992f3dceea36c6b35ca74e68113e2
Sep 08 00:39:22.121652 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:22.121586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv" event={"ID":"99e916f6-825c-4858-9882-d7670a697d5b","Type":"ContainerStarted","Data":"441491bd2bf7e087745ce4d337f29ca3cb5992f3dceea36c6b35ca74e68113e2"}
Sep 08 00:39:23.337334 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.337304    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-85d49875c6-hfqzv"]
Sep 08 00:39:23.340950 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.340929    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:23.345045 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.345023    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 08 00:39:23.345045 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.345037    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gs4qj\""
Sep 08 00:39:23.361586 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.361562    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-85d49875c6-hfqzv"]
Sep 08 00:39:23.444281 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.444252    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-bb7b75cc4-w4v82"]
Sep 08 00:39:23.447658 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.447630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.452050 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.451985    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 08 00:39:23.452672 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.452651    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Sep 08 00:39:23.452763 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.452694    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gqcth\""
Sep 08 00:39:23.475529 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.475503    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77d88\" (UniqueName: \"kubernetes.io/projected/da04b547-df71-4029-b7ce-1b825d394d50-kube-api-access-77d88\") pod \"kserve-controller-manager-85d49875c6-hfqzv\" (UID: \"da04b547-df71-4029-b7ce-1b825d394d50\") " pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:23.475657 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.475582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04b547-df71-4029-b7ce-1b825d394d50-cert\") pod \"kserve-controller-manager-85d49875c6-hfqzv\" (UID: \"da04b547-df71-4029-b7ce-1b825d394d50\") " pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:23.490078 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.490051    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-bb7b75cc4-w4v82"]
Sep 08 00:39:23.576257 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.576222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77d88\" (UniqueName: \"kubernetes.io/projected/da04b547-df71-4029-b7ce-1b825d394d50-kube-api-access-77d88\") pod \"kserve-controller-manager-85d49875c6-hfqzv\" (UID: \"da04b547-df71-4029-b7ce-1b825d394d50\") " pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:23.576257 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.576261    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-metrics-certs\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.576485 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.576308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-cert\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.576485 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.576327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxjn\" (UniqueName: \"kubernetes.io/projected/8ae88f22-e7ce-43c7-80f3-b25681d5f292-kube-api-access-ztxjn\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.576485 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.576400    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04b547-df71-4029-b7ce-1b825d394d50-cert\") pod \"kserve-controller-manager-85d49875c6-hfqzv\" (UID: \"da04b547-df71-4029-b7ce-1b825d394d50\") " pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:23.576668 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:39:23.576511    2555 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 08 00:39:23.576668 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:39:23.576563    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da04b547-df71-4029-b7ce-1b825d394d50-cert podName:da04b547-df71-4029-b7ce-1b825d394d50 nodeName:}" failed. No retries permitted until 2026-09-08 00:39:24.076549671 +0000 UTC m=+398.419852669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da04b547-df71-4029-b7ce-1b825d394d50-cert") pod "kserve-controller-manager-85d49875c6-hfqzv" (UID: "da04b547-df71-4029-b7ce-1b825d394d50") : secret "kserve-webhook-server-cert" not found
Sep 08 00:39:23.589166 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.589142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77d88\" (UniqueName: \"kubernetes.io/projected/da04b547-df71-4029-b7ce-1b825d394d50-kube-api-access-77d88\") pod \"kserve-controller-manager-85d49875c6-hfqzv\" (UID: \"da04b547-df71-4029-b7ce-1b825d394d50\") " pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:23.677769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.677745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-metrics-certs\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.677886 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.677788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-cert\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.677886 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.677806    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxjn\" (UniqueName: \"kubernetes.io/projected/8ae88f22-e7ce-43c7-80f3-b25681d5f292-kube-api-access-ztxjn\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.677990 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:39:23.677895    2555 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 08 00:39:23.677990 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:39:23.677959    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-metrics-certs podName:8ae88f22-e7ce-43c7-80f3-b25681d5f292 nodeName:}" failed. No retries permitted until 2026-09-08 00:39:24.177940777 +0000 UTC m=+398.521243793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-metrics-certs") pod "odh-model-controller-bb7b75cc4-w4v82" (UID: "8ae88f22-e7ce-43c7-80f3-b25681d5f292") : secret "odh-model-controller-metrics-tls" not found
Sep 08 00:39:23.680280 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.680259    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-cert\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:23.693919 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:23.693892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxjn\" (UniqueName: \"kubernetes.io/projected/8ae88f22-e7ce-43c7-80f3-b25681d5f292-kube-api-access-ztxjn\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:24.082129 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.082091    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04b547-df71-4029-b7ce-1b825d394d50-cert\") pod \"kserve-controller-manager-85d49875c6-hfqzv\" (UID: \"da04b547-df71-4029-b7ce-1b825d394d50\") " pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:24.084607 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.084571    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04b547-df71-4029-b7ce-1b825d394d50-cert\") pod \"kserve-controller-manager-85d49875c6-hfqzv\" (UID: \"da04b547-df71-4029-b7ce-1b825d394d50\") " pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:24.135113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.135083    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv" event={"ID":"99e916f6-825c-4858-9882-d7670a697d5b","Type":"ContainerStarted","Data":"9fe309c67a91f773d73fb05ba75f0d99846f3f82b5e2801c9acd5b8042f046a2"}
Sep 08 00:39:24.135235 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.135146    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:24.167633 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.167573    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv" podStartSLOduration=2.363973083 podStartE2EDuration="4.167557969s" totalImagesPullingTime="1.803584886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:24.166108069 +0000 UTC m=+398.509411088" watchObservedRunningTime="2026-09-08 00:39:24.167557969 +0000 UTC m=+398.510860988"
Sep 08 00:39:24.182568 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.182535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-metrics-certs\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:24.184924 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.184900    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae88f22-e7ce-43c7-80f3-b25681d5f292-metrics-certs\") pod \"odh-model-controller-bb7b75cc4-w4v82\" (UID: \"8ae88f22-e7ce-43c7-80f3-b25681d5f292\") " pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:24.250889 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.250864    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:24.357994 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.357931    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:24.524326 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.524299    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-85d49875c6-hfqzv"]
Sep 08 00:39:24.525780 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:39:24.525750    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04b547_df71_4029_b7ce_1b825d394d50.slice/crio-d1f2e4e30d9d86b33129222c165db8926c0fb3b440c77405debc0dca0e7c7d43 WatchSource:0}: Error finding container d1f2e4e30d9d86b33129222c165db8926c0fb3b440c77405debc0dca0e7c7d43: Status 404 returned error can't find the container with id d1f2e4e30d9d86b33129222c165db8926c0fb3b440c77405debc0dca0e7c7d43
Sep 08 00:39:24.633436 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:24.633338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-bb7b75cc4-w4v82"]
Sep 08 00:39:24.635543 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:39:24.635514    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae88f22_e7ce_43c7_80f3_b25681d5f292.slice/crio-82b53753159ad6dee18a6883fd32cdec444450b1dad37c2de447b79b63865bb8 WatchSource:0}: Error finding container 82b53753159ad6dee18a6883fd32cdec444450b1dad37c2de447b79b63865bb8: Status 404 returned error can't find the container with id 82b53753159ad6dee18a6883fd32cdec444450b1dad37c2de447b79b63865bb8
Sep 08 00:39:25.141928 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:25.141890    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82" event={"ID":"8ae88f22-e7ce-43c7-80f3-b25681d5f292","Type":"ContainerStarted","Data":"82b53753159ad6dee18a6883fd32cdec444450b1dad37c2de447b79b63865bb8"}
Sep 08 00:39:25.143119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:25.143092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv" event={"ID":"da04b547-df71-4029-b7ce-1b825d394d50","Type":"ContainerStarted","Data":"d1f2e4e30d9d86b33129222c165db8926c0fb3b440c77405debc0dca0e7c7d43"}
Sep 08 00:39:28.165913 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:28.165815    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82" event={"ID":"8ae88f22-e7ce-43c7-80f3-b25681d5f292","Type":"ContainerStarted","Data":"4d4097a721a21f323afbae10aba2c00186bbe125d83c5fbfe5ced01318d2c861"}
Sep 08 00:39:28.166293 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:28.165955    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:28.167472 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:28.167447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv" event={"ID":"da04b547-df71-4029-b7ce-1b825d394d50","Type":"ContainerStarted","Data":"377a9a29d4656927bde73245b262a3172b3d8aa40d12f418b20a144fea517c2c"}
Sep 08 00:39:28.167616 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:28.167563    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:39:28.185483 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:28.185428    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82" podStartSLOduration=1.9406942329999999 podStartE2EDuration="5.185408306s" totalImagesPullingTime="3.244714073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:28.182111469 +0000 UTC m=+402.525414488" watchObservedRunningTime="2026-09-08 00:39:28.185408306 +0000 UTC m=+402.528711325"
Sep 08 00:39:28.199951 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:28.199901    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv" podStartSLOduration=1.84639904 podStartE2EDuration="5.199886815s" totalImagesPullingTime="3.353487775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:28.198044109 +0000 UTC m=+402.541347129" watchObservedRunningTime="2026-09-08 00:39:28.199886815 +0000 UTC m=+402.543189834"
Sep 08 00:39:29.521392 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.521361    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xblfb"]
Sep 08 00:39:29.524943 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.524924    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xblfb"
Sep 08 00:39:29.537439 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.537419    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xblfb"]
Sep 08 00:39:29.629975 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.629945    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2pd\" (UniqueName: \"kubernetes.io/projected/5d2cb01d-223e-41e3-91e5-4eb43658379b-kube-api-access-4v2pd\") pod \"authorino-5dc7ccb5f-xblfb\" (UID: \"5d2cb01d-223e-41e3-91e5-4eb43658379b\") " pod="kuadrant-system/authorino-5dc7ccb5f-xblfb"
Sep 08 00:39:29.727503 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.727475    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xblfb"]
Sep 08 00:39:29.727696 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:39:29.727677    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4v2pd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-xblfb" podUID="5d2cb01d-223e-41e3-91e5-4eb43658379b"
Sep 08 00:39:29.730724 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.730697    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2pd\" (UniqueName: \"kubernetes.io/projected/5d2cb01d-223e-41e3-91e5-4eb43658379b-kube-api-access-4v2pd\") pod \"authorino-5dc7ccb5f-xblfb\" (UID: \"5d2cb01d-223e-41e3-91e5-4eb43658379b\") " pod="kuadrant-system/authorino-5dc7ccb5f-xblfb"
Sep 08 00:39:29.739547 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.739526    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2pd\" (UniqueName: \"kubernetes.io/projected/5d2cb01d-223e-41e3-91e5-4eb43658379b-kube-api-access-4v2pd\") pod \"authorino-5dc7ccb5f-xblfb\" (UID: \"5d2cb01d-223e-41e3-91e5-4eb43658379b\") " pod="kuadrant-system/authorino-5dc7ccb5f-xblfb"
Sep 08 00:39:29.753131 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.753108    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cd68656b8-sd8ln"]
Sep 08 00:39:29.756657 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.756640    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd68656b8-sd8ln"
Sep 08 00:39:29.763094 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.763071    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd68656b8-sd8ln"]
Sep 08 00:39:29.831566 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.831528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698g2\" (UniqueName: \"kubernetes.io/projected/3d7c7eb0-6d9e-447d-b264-a652c156b5ae-kube-api-access-698g2\") pod \"authorino-6cd68656b8-sd8ln\" (UID: \"3d7c7eb0-6d9e-447d-b264-a652c156b5ae\") " pod="kuadrant-system/authorino-6cd68656b8-sd8ln"
Sep 08 00:39:29.931969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.931938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698g2\" (UniqueName: \"kubernetes.io/projected/3d7c7eb0-6d9e-447d-b264-a652c156b5ae-kube-api-access-698g2\") pod \"authorino-6cd68656b8-sd8ln\" (UID: \"3d7c7eb0-6d9e-447d-b264-a652c156b5ae\") " pod="kuadrant-system/authorino-6cd68656b8-sd8ln"
Sep 08 00:39:29.943045 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:29.943016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-698g2\" (UniqueName: \"kubernetes.io/projected/3d7c7eb0-6d9e-447d-b264-a652c156b5ae-kube-api-access-698g2\") pod \"authorino-6cd68656b8-sd8ln\" (UID: \"3d7c7eb0-6d9e-447d-b264-a652c156b5ae\") " pod="kuadrant-system/authorino-6cd68656b8-sd8ln"
Sep 08 00:39:30.015245 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.015219    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cd68656b8-sd8ln"]
Sep 08 00:39:30.015455 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.015444    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd68656b8-sd8ln"
Sep 08 00:39:30.043416 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.043379    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7ddb857b4b-dnh2b"]
Sep 08 00:39:30.048705 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.048681    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.051615 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.051571    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 00:39:30.052695 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.052673    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ddb857b4b-dnh2b"]
Sep 08 00:39:30.134069 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.133970    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mstvw\" (UniqueName: \"kubernetes.io/projected/0e6c027d-1196-48af-8453-d76db899a053-kube-api-access-mstvw\") pod \"authorino-7ddb857b4b-dnh2b\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") " pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.134069 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.134036    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0e6c027d-1196-48af-8453-d76db899a053-tls-cert\") pod \"authorino-7ddb857b4b-dnh2b\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") " pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.177136 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.177107    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xblfb"
Sep 08 00:39:30.184812 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.184793    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xblfb"
Sep 08 00:39:30.237612 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.237117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mstvw\" (UniqueName: \"kubernetes.io/projected/0e6c027d-1196-48af-8453-d76db899a053-kube-api-access-mstvw\") pod \"authorino-7ddb857b4b-dnh2b\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") " pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.237612 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.237270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0e6c027d-1196-48af-8453-d76db899a053-tls-cert\") pod \"authorino-7ddb857b4b-dnh2b\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") " pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.249154 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.241898    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0e6c027d-1196-48af-8453-d76db899a053-tls-cert\") pod \"authorino-7ddb857b4b-dnh2b\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") " pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.251328 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.249644    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mstvw\" (UniqueName: \"kubernetes.io/projected/0e6c027d-1196-48af-8453-d76db899a053-kube-api-access-mstvw\") pod \"authorino-7ddb857b4b-dnh2b\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") " pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.310833 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.310810    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cd68656b8-sd8ln"]
Sep 08 00:39:30.313459 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:39:30.313435    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d7c7eb0_6d9e_447d_b264_a652c156b5ae.slice/crio-810b908282ede7dd45127f97548ebb2bbf12339ef85c2c779a5c1b29005531a4 WatchSource:0}: Error finding container 810b908282ede7dd45127f97548ebb2bbf12339ef85c2c779a5c1b29005531a4: Status 404 returned error can't find the container with id 810b908282ede7dd45127f97548ebb2bbf12339ef85c2c779a5c1b29005531a4
Sep 08 00:39:30.337969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.337943    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v2pd\" (UniqueName: \"kubernetes.io/projected/5d2cb01d-223e-41e3-91e5-4eb43658379b-kube-api-access-4v2pd\") pod \"5d2cb01d-223e-41e3-91e5-4eb43658379b\" (UID: \"5d2cb01d-223e-41e3-91e5-4eb43658379b\") "
Sep 08 00:39:30.339749 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.339725    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2cb01d-223e-41e3-91e5-4eb43658379b-kube-api-access-4v2pd" (OuterVolumeSpecName: "kube-api-access-4v2pd") pod "5d2cb01d-223e-41e3-91e5-4eb43658379b" (UID: "5d2cb01d-223e-41e3-91e5-4eb43658379b"). InnerVolumeSpecName "kube-api-access-4v2pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:39:30.359619 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.359585    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:39:30.438875 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.438841    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v2pd\" (UniqueName: \"kubernetes.io/projected/5d2cb01d-223e-41e3-91e5-4eb43658379b-kube-api-access-4v2pd\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:39:30.630715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.630690    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ddb857b4b-dnh2b"]
Sep 08 00:39:30.632331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:39:30.632303    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e6c027d_1196_48af_8453_d76db899a053.slice/crio-fcfb2775007cb99e6a1f8372d84fbd2838f0375891ec374a300a39cd43638ed8 WatchSource:0}: Error finding container fcfb2775007cb99e6a1f8372d84fbd2838f0375891ec374a300a39cd43638ed8: Status 404 returned error can't find the container with id fcfb2775007cb99e6a1f8372d84fbd2838f0375891ec374a300a39cd43638ed8
Sep 08 00:39:30.724302 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:30.724247    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"
Sep 08 00:39:31.185377 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.185330    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b" event={"ID":"0e6c027d-1196-48af-8453-d76db899a053","Type":"ContainerStarted","Data":"4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb"}
Sep 08 00:39:31.185500 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.185385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b" event={"ID":"0e6c027d-1196-48af-8453-d76db899a053","Type":"ContainerStarted","Data":"fcfb2775007cb99e6a1f8372d84fbd2838f0375891ec374a300a39cd43638ed8"}
Sep 08 00:39:31.187732 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.187701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd68656b8-sd8ln" event={"ID":"3d7c7eb0-6d9e-447d-b264-a652c156b5ae","Type":"ContainerStarted","Data":"6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3"}
Sep 08 00:39:31.187848 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.187731    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xblfb"
Sep 08 00:39:31.187848 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.187736    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cd68656b8-sd8ln" podUID="3d7c7eb0-6d9e-447d-b264-a652c156b5ae" containerName="authorino" containerID="cri-o://6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3" gracePeriod=30
Sep 08 00:39:31.187954 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.187747    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd68656b8-sd8ln" event={"ID":"3d7c7eb0-6d9e-447d-b264-a652c156b5ae","Type":"ContainerStarted","Data":"810b908282ede7dd45127f97548ebb2bbf12339ef85c2c779a5c1b29005531a4"}
Sep 08 00:39:31.203213 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.203172    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b" podStartSLOduration=0.739121173 podStartE2EDuration="1.203158615s" totalImagesPullingTime="464.037442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:31.201965183 +0000 UTC m=+405.545268207" watchObservedRunningTime="2026-09-08 00:39:31.203158615 +0000 UTC m=+405.546461634"
Sep 08 00:39:31.220670 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.220541    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cd68656b8-sd8ln" podStartSLOduration=1.4500436620000001 podStartE2EDuration="2.220520705s" totalImagesPullingTime="770.477043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:31.219314703 +0000 UTC m=+405.562617723" watchObservedRunningTime="2026-09-08 00:39:31.220520705 +0000 UTC m=+405.563823725"
Sep 08 00:39:31.236968 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.236930    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fwsn6"]
Sep 08 00:39:31.237340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.237301    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-fwsn6" podUID="353d3cb0-e64d-4ee3-b35c-1354849c5598" containerName="authorino" containerID="cri-o://4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c" gracePeriod=30
Sep 08 00:39:31.297548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.297518    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xblfb"]
Sep 08 00:39:31.317737 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.317698    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xblfb"]
Sep 08 00:39:31.475627 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.475602    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd68656b8-sd8ln"
Sep 08 00:39:31.517615 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.517579    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fwsn6"
Sep 08 00:39:31.652577 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.652483    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-698g2\" (UniqueName: \"kubernetes.io/projected/3d7c7eb0-6d9e-447d-b264-a652c156b5ae-kube-api-access-698g2\") pod \"3d7c7eb0-6d9e-447d-b264-a652c156b5ae\" (UID: \"3d7c7eb0-6d9e-447d-b264-a652c156b5ae\") "
Sep 08 00:39:31.653046 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.652583    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ngtc\" (UniqueName: \"kubernetes.io/projected/353d3cb0-e64d-4ee3-b35c-1354849c5598-kube-api-access-8ngtc\") pod \"353d3cb0-e64d-4ee3-b35c-1354849c5598\" (UID: \"353d3cb0-e64d-4ee3-b35c-1354849c5598\") "
Sep 08 00:39:31.654784 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.654755    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7c7eb0-6d9e-447d-b264-a652c156b5ae-kube-api-access-698g2" (OuterVolumeSpecName: "kube-api-access-698g2") pod "3d7c7eb0-6d9e-447d-b264-a652c156b5ae" (UID: "3d7c7eb0-6d9e-447d-b264-a652c156b5ae"). InnerVolumeSpecName "kube-api-access-698g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:39:31.655170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.655148    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353d3cb0-e64d-4ee3-b35c-1354849c5598-kube-api-access-8ngtc" (OuterVolumeSpecName: "kube-api-access-8ngtc") pod "353d3cb0-e64d-4ee3-b35c-1354849c5598" (UID: "353d3cb0-e64d-4ee3-b35c-1354849c5598"). InnerVolumeSpecName "kube-api-access-8ngtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:39:31.753618 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.753573    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-698g2\" (UniqueName: \"kubernetes.io/projected/3d7c7eb0-6d9e-447d-b264-a652c156b5ae-kube-api-access-698g2\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:39:31.753618 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:31.753612    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngtc\" (UniqueName: \"kubernetes.io/projected/353d3cb0-e64d-4ee3-b35c-1354849c5598-kube-api-access-8ngtc\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:39:32.154318 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.154287    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2cb01d-223e-41e3-91e5-4eb43658379b" path="/var/lib/kubelet/pods/5d2cb01d-223e-41e3-91e5-4eb43658379b/volumes"
Sep 08 00:39:32.194448 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.194420    2555 generic.go:356] "Generic (PLEG): container finished" podID="353d3cb0-e64d-4ee3-b35c-1354849c5598" containerID="4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c" exitCode=0
Sep 08 00:39:32.194560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.194478    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-fwsn6"
Sep 08 00:39:32.194560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.194521    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fwsn6" event={"ID":"353d3cb0-e64d-4ee3-b35c-1354849c5598","Type":"ContainerDied","Data":"4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c"}
Sep 08 00:39:32.194560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.194552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-fwsn6" event={"ID":"353d3cb0-e64d-4ee3-b35c-1354849c5598","Type":"ContainerDied","Data":"c05a014cd2b938e9d90c27410509209ff721e01ad64846df10e23798575914ad"}
Sep 08 00:39:32.194786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.194573    2555 scope.go:117] "RemoveContainer" containerID="4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c"
Sep 08 00:39:32.196037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.196006    2555 generic.go:356] "Generic (PLEG): container finished" podID="3d7c7eb0-6d9e-447d-b264-a652c156b5ae" containerID="6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3" exitCode=2
Sep 08 00:39:32.196137 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.196059    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd68656b8-sd8ln"
Sep 08 00:39:32.196203 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.196171    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd68656b8-sd8ln" event={"ID":"3d7c7eb0-6d9e-447d-b264-a652c156b5ae","Type":"ContainerDied","Data":"6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3"}
Sep 08 00:39:32.196264 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.196216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd68656b8-sd8ln" event={"ID":"3d7c7eb0-6d9e-447d-b264-a652c156b5ae","Type":"ContainerDied","Data":"810b908282ede7dd45127f97548ebb2bbf12339ef85c2c779a5c1b29005531a4"}
Sep 08 00:39:32.203709 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.203689    2555 scope.go:117] "RemoveContainer" containerID="4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c"
Sep 08 00:39:32.203984 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:39:32.203964    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c\": container with ID starting with 4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c not found: ID does not exist" containerID="4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c"
Sep 08 00:39:32.204037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.203991    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c"} err="failed to get container status \"4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c\": rpc error: code = NotFound desc = could not find container \"4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c\": container with ID starting with 4c45fe00906c2e7fa23d4bd086c1e00fbaee6b4b73ec013e31d6d16c99ab8e7c not found: ID does not exist"
Sep 08 00:39:32.204037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.204008    2555 scope.go:117] "RemoveContainer" containerID="6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3"
Sep 08 00:39:32.211138 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.211118    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cd68656b8-sd8ln"]
Sep 08 00:39:32.213284 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.213261    2555 scope.go:117] "RemoveContainer" containerID="6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3"
Sep 08 00:39:32.213622 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:39:32.213577    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3\": container with ID starting with 6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3 not found: ID does not exist" containerID="6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3"
Sep 08 00:39:32.213686 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.213632    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3"} err="failed to get container status \"6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3\": rpc error: code = NotFound desc = could not find container \"6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3\": container with ID starting with 6ca69364cd37ce2c2908854dd3a0294871eba457c6f98bedb216e7e5a166fcf3 not found: ID does not exist"
Sep 08 00:39:32.214715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.214699    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cd68656b8-sd8ln"]
Sep 08 00:39:32.226150 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.226129    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fwsn6"]
Sep 08 00:39:32.229476 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:32.229457    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-fwsn6"]
Sep 08 00:39:34.152762 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:34.152731    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353d3cb0-e64d-4ee3-b35c-1354849c5598" path="/var/lib/kubelet/pods/353d3cb0-e64d-4ee3-b35c-1354849c5598/volumes"
Sep 08 00:39:34.153124 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:34.153066    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7c7eb0-6d9e-447d-b264-a652c156b5ae" path="/var/lib/kubelet/pods/3d7c7eb0-6d9e-447d-b264-a652c156b5ae/volumes"
Sep 08 00:39:35.146228 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:35.146195    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-tw9lv"
Sep 08 00:39:39.175290 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:39.175252    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-bb7b75cc4-w4v82"
Sep 08 00:39:59.178560 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:39:59.178525    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-85d49875c6-hfqzv"
Sep 08 00:40:44.445588 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:40:44.445546    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:40:52.331989 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:40:52.331950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:41:03.639957 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:03.639920    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:41:11.937615 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:11.937556    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:41:16.932675 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.932643    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"]
Sep 08 00:41:16.933224 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.933203    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353d3cb0-e64d-4ee3-b35c-1354849c5598" containerName="authorino"
Sep 08 00:41:16.933309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.933227    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="353d3cb0-e64d-4ee3-b35c-1354849c5598" containerName="authorino"
Sep 08 00:41:16.933309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.933278    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7c7eb0-6d9e-447d-b264-a652c156b5ae" containerName="authorino"
Sep 08 00:41:16.933309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.933287    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7c7eb0-6d9e-447d-b264-a652c156b5ae" containerName="authorino"
Sep 08 00:41:16.933468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.933368    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="353d3cb0-e64d-4ee3-b35c-1354849c5598" containerName="authorino"
Sep 08 00:41:16.933468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.933381    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7c7eb0-6d9e-447d-b264-a652c156b5ae" containerName="authorino"
Sep 08 00:41:16.936413 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.936393    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:16.940543 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.940521    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cnbvb\""
Sep 08 00:41:16.940684 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.940637    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 00:41:16.940915 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.940890    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 00:41:16.941027 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.940966    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 00:41:16.953080 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:16.953058    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"]
Sep 08 00:41:17.070813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.070783    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.070813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.070817    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.071000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.070881    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/100eeba4-5850-4215-acf4-e73d0e7efcdc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.071000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.070912    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5r9z\" (UniqueName: \"kubernetes.io/projected/100eeba4-5850-4215-acf4-e73d0e7efcdc-kube-api-access-g5r9z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.071000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.070933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.071000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.070961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.171889 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.171847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/100eeba4-5850-4215-acf4-e73d0e7efcdc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.171889 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.171888    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5r9z\" (UniqueName: \"kubernetes.io/projected/100eeba4-5850-4215-acf4-e73d0e7efcdc-kube-api-access-g5r9z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.172093 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.171907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.172093 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.171976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.172093 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.172067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.172222 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.172091    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.172326 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.172306    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.172399 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.172366    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.172512 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.172455    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.174137 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.174118    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/100eeba4-5850-4215-acf4-e73d0e7efcdc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.174447 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.174431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/100eeba4-5850-4215-acf4-e73d0e7efcdc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.181341 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.181315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5r9z\" (UniqueName: \"kubernetes.io/projected/100eeba4-5850-4215-acf4-e73d0e7efcdc-kube-api-access-g5r9z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85\" (UID: \"100eeba4-5850-4215-acf4-e73d0e7efcdc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.247112 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.247022    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:17.520013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.519927    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"]
Sep 08 00:41:17.523643 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:41:17.523612    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100eeba4_5850_4215_acf4_e73d0e7efcdc.slice/crio-07897307968d5d2ad3e3bc11147baa9e633620a9e082a190c6d9d30aec697c58 WatchSource:0}: Error finding container 07897307968d5d2ad3e3bc11147baa9e633620a9e082a190c6d9d30aec697c58: Status 404 returned error can't find the container with id 07897307968d5d2ad3e3bc11147baa9e633620a9e082a190c6d9d30aec697c58
Sep 08 00:41:17.757320 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:17.757279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85" event={"ID":"100eeba4-5850-4215-acf4-e73d0e7efcdc","Type":"ContainerStarted","Data":"07897307968d5d2ad3e3bc11147baa9e633620a9e082a190c6d9d30aec697c58"}
Sep 08 00:41:18.745121 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:18.745034    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:41:18.765116 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:18.765077    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85" event={"ID":"100eeba4-5850-4215-acf4-e73d0e7efcdc","Type":"ContainerStarted","Data":"eae35f3ac282396491a6ec565ad051c21418f9f8cdecd9e6a5476d49686569ca"}
Sep 08 00:41:18.785435 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:18.785378    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85" podStartSLOduration=1.816158443 podStartE2EDuration="2.785362555s" totalImagesPullingTime="969.204112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:41:18.783851993 +0000 UTC m=+513.127155013" watchObservedRunningTime="2026-09-08 00:41:18.785362555 +0000 UTC m=+513.128665622"
Sep 08 00:41:19.247380 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:19.247340    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:27.248066 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:27.248019    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:27.260831 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:27.260804    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:27.820412 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:27.820384    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85"
Sep 08 00:41:36.937120 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:36.937083    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:41:49.537614 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:41:49.537549    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:42:46.107903 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:42:46.107874    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:42:46.109332 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:42:46.109307    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:43:01.466951 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.466915    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54784f667-kf4l5"]
Sep 08 00:43:01.470422 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.470406    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:01.477004 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.476980    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54784f667-kf4l5"]
Sep 08 00:43:01.613046 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.613015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkp9\" (UniqueName: \"kubernetes.io/projected/3e41b3fe-7e50-4d2e-a55f-764638857b7f-kube-api-access-wjkp9\") pod \"authorino-54784f667-kf4l5\" (UID: \"3e41b3fe-7e50-4d2e-a55f-764638857b7f\") " pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:01.613184 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.613162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e41b3fe-7e50-4d2e-a55f-764638857b7f-tls-cert\") pod \"authorino-54784f667-kf4l5\" (UID: \"3e41b3fe-7e50-4d2e-a55f-764638857b7f\") " pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:01.714294 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.714269    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e41b3fe-7e50-4d2e-a55f-764638857b7f-tls-cert\") pod \"authorino-54784f667-kf4l5\" (UID: \"3e41b3fe-7e50-4d2e-a55f-764638857b7f\") " pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:01.714414 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.714308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkp9\" (UniqueName: \"kubernetes.io/projected/3e41b3fe-7e50-4d2e-a55f-764638857b7f-kube-api-access-wjkp9\") pod \"authorino-54784f667-kf4l5\" (UID: \"3e41b3fe-7e50-4d2e-a55f-764638857b7f\") " pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:01.716745 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.716717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3e41b3fe-7e50-4d2e-a55f-764638857b7f-tls-cert\") pod \"authorino-54784f667-kf4l5\" (UID: \"3e41b3fe-7e50-4d2e-a55f-764638857b7f\") " pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:01.723252 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.723207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkp9\" (UniqueName: \"kubernetes.io/projected/3e41b3fe-7e50-4d2e-a55f-764638857b7f-kube-api-access-wjkp9\") pod \"authorino-54784f667-kf4l5\" (UID: \"3e41b3fe-7e50-4d2e-a55f-764638857b7f\") " pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:01.780050 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:01.780027    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54784f667-kf4l5"
Sep 08 00:43:02.053477 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:02.053406    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54784f667-kf4l5"]
Sep 08 00:43:02.055732 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:02.055702    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e41b3fe_7e50_4d2e_a55f_764638857b7f.slice/crio-e5a91cf06df6a8a6388b8c1de147438aac3d760c42cf4016efc04dfc692af62b WatchSource:0}: Error finding container e5a91cf06df6a8a6388b8c1de147438aac3d760c42cf4016efc04dfc692af62b: Status 404 returned error can't find the container with id e5a91cf06df6a8a6388b8c1de147438aac3d760c42cf4016efc04dfc692af62b
Sep 08 00:43:02.057164 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:02.057145    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:43:02.308315 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:02.308232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54784f667-kf4l5" event={"ID":"3e41b3fe-7e50-4d2e-a55f-764638857b7f","Type":"ContainerStarted","Data":"e5a91cf06df6a8a6388b8c1de147438aac3d760c42cf4016efc04dfc692af62b"}
Sep 08 00:43:03.316249 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.316215    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54784f667-kf4l5" event={"ID":"3e41b3fe-7e50-4d2e-a55f-764638857b7f","Type":"ContainerStarted","Data":"1bba702747dbfa766f6f52b4a3c24fffd55a5627f9189816cec2aca24012b505"}
Sep 08 00:43:03.335111 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.335058    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54784f667-kf4l5" podStartSLOduration=1.505860459 podStartE2EDuration="2.335042333s" totalImagesPullingTime="829.181874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:03.332713797 +0000 UTC m=+617.676016841" watchObservedRunningTime="2026-09-08 00:43:03.335042333 +0000 UTC m=+617.678345351"
Sep 08 00:43:03.361097 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.361060    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ddb857b4b-dnh2b"]
Sep 08 00:43:03.361460 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.361267    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b" podUID="0e6c027d-1196-48af-8453-d76db899a053" containerName="authorino" containerID="cri-o://4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb" gracePeriod=30
Sep 08 00:43:03.621328 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.621300    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:43:03.633571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.633545    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0e6c027d-1196-48af-8453-d76db899a053-tls-cert\") pod \"0e6c027d-1196-48af-8453-d76db899a053\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") "
Sep 08 00:43:03.633719 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.633584    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mstvw\" (UniqueName: \"kubernetes.io/projected/0e6c027d-1196-48af-8453-d76db899a053-kube-api-access-mstvw\") pod \"0e6c027d-1196-48af-8453-d76db899a053\" (UID: \"0e6c027d-1196-48af-8453-d76db899a053\") "
Sep 08 00:43:03.635567 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.635538    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6c027d-1196-48af-8453-d76db899a053-kube-api-access-mstvw" (OuterVolumeSpecName: "kube-api-access-mstvw") pod "0e6c027d-1196-48af-8453-d76db899a053" (UID: "0e6c027d-1196-48af-8453-d76db899a053"). InnerVolumeSpecName "kube-api-access-mstvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:43:03.643985 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.643963    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6c027d-1196-48af-8453-d76db899a053-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0e6c027d-1196-48af-8453-d76db899a053" (UID: "0e6c027d-1196-48af-8453-d76db899a053"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:43:03.734526 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.734484    2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0e6c027d-1196-48af-8453-d76db899a053-tls-cert\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:43:03.734526 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:03.734520    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mstvw\" (UniqueName: \"kubernetes.io/projected/0e6c027d-1196-48af-8453-d76db899a053-kube-api-access-mstvw\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:43:04.323252 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.323213    2555 generic.go:356] "Generic (PLEG): container finished" podID="0e6c027d-1196-48af-8453-d76db899a053" containerID="4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb" exitCode=0
Sep 08 00:43:04.323646 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.323269    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b"
Sep 08 00:43:04.323646 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.323313    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b" event={"ID":"0e6c027d-1196-48af-8453-d76db899a053","Type":"ContainerDied","Data":"4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb"}
Sep 08 00:43:04.323646 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.323363    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ddb857b4b-dnh2b" event={"ID":"0e6c027d-1196-48af-8453-d76db899a053","Type":"ContainerDied","Data":"fcfb2775007cb99e6a1f8372d84fbd2838f0375891ec374a300a39cd43638ed8"}
Sep 08 00:43:04.323646 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.323389    2555 scope.go:117] "RemoveContainer" containerID="4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb"
Sep 08 00:43:04.332334 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.332318    2555 scope.go:117] "RemoveContainer" containerID="4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb"
Sep 08 00:43:04.332576 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:43:04.332555    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb\": container with ID starting with 4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb not found: ID does not exist" containerID="4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb"
Sep 08 00:43:04.332651 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.332583    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb"} err="failed to get container status \"4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb\": rpc error: code = NotFound desc = could not find container \"4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb\": container with ID starting with 4d06de3c792d1e4b9d21e07752b562b4100bec12857d83a35c9c5639f3cf6ccb not found: ID does not exist"
Sep 08 00:43:04.340480 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.340458    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7ddb857b4b-dnh2b"]
Sep 08 00:43:04.346316 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:04.346294    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7ddb857b4b-dnh2b"]
Sep 08 00:43:06.155003 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:06.154963    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6c027d-1196-48af-8453-d76db899a053" path="/var/lib/kubelet/pods/0e6c027d-1196-48af-8453-d76db899a053/volumes"
Sep 08 00:43:22.343711 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.343630    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:22.344070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.344054    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6c027d-1196-48af-8453-d76db899a053" containerName="authorino"
Sep 08 00:43:22.344070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.344071    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6c027d-1196-48af-8453-d76db899a053" containerName="authorino"
Sep 08 00:43:22.344163 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.344130    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6c027d-1196-48af-8453-d76db899a053" containerName="authorino"
Sep 08 00:43:22.347108 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.347090    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:22.349926 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.349903    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fj6d7\""
Sep 08 00:43:22.351936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.351904    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:22.395227 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.395203    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58zw\" (UniqueName: \"kubernetes.io/projected/c489a05b-bf44-4b3c-a2ce-830e78d956fe-kube-api-access-m58zw\") pod \"test-curl-5884-140284875727792\" (UID: \"c489a05b-bf44-4b3c-a2ce-830e78d956fe\") " pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:22.496684 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.496649    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58zw\" (UniqueName: \"kubernetes.io/projected/c489a05b-bf44-4b3c-a2ce-830e78d956fe-kube-api-access-m58zw\") pod \"test-curl-5884-140284875727792\" (UID: \"c489a05b-bf44-4b3c-a2ce-830e78d956fe\") " pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:22.510949 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.510926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58zw\" (UniqueName: \"kubernetes.io/projected/c489a05b-bf44-4b3c-a2ce-830e78d956fe-kube-api-access-m58zw\") pod \"test-curl-5884-140284875727792\" (UID: \"c489a05b-bf44-4b3c-a2ce-830e78d956fe\") " pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:22.657541 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.657470    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:22.928678 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:22.928645    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc489a05b_bf44_4b3c_a2ce_830e78d956fe.slice/crio-a12bc1ffc6bf563304c2c175369f115ad144e7f45627f99c6290b1cc7ec7942a WatchSource:0}: Error finding container a12bc1ffc6bf563304c2c175369f115ad144e7f45627f99c6290b1cc7ec7942a: Status 404 returned error can't find the container with id a12bc1ffc6bf563304c2c175369f115ad144e7f45627f99c6290b1cc7ec7942a
Sep 08 00:43:22.929833 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:22.929807    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:23.424203 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:23.424170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5884-140284875727792" event={"ID":"c489a05b-bf44-4b3c-a2ce-830e78d956fe","Type":"ContainerStarted","Data":"a12bc1ffc6bf563304c2c175369f115ad144e7f45627f99c6290b1cc7ec7942a"}
Sep 08 00:43:24.431209 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:24.431173    2555 generic.go:356] "Generic (PLEG): container finished" podID="c489a05b-bf44-4b3c-a2ce-830e78d956fe" containerID="15057b91b91e461a6f1ba9559639aecbeb644e2b83819dab65e3a46a698186bb" exitCode=0
Sep 08 00:43:24.431583 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:24.431223    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5884-140284875727792" event={"ID":"c489a05b-bf44-4b3c-a2ce-830e78d956fe","Type":"ContainerDied","Data":"15057b91b91e461a6f1ba9559639aecbeb644e2b83819dab65e3a46a698186bb"}
Sep 08 00:43:25.575260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.575240    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:25.613935 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.613906    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-5884-140284875727792_c489a05b-bf44-4b3c-a2ce-830e78d956fe/test-curl-5884-140284875727792/0.log"
Sep 08 00:43:25.621135 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.621115    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m58zw\" (UniqueName: \"kubernetes.io/projected/c489a05b-bf44-4b3c-a2ce-830e78d956fe-kube-api-access-m58zw\") pod \"c489a05b-bf44-4b3c-a2ce-830e78d956fe\" (UID: \"c489a05b-bf44-4b3c-a2ce-830e78d956fe\") "
Sep 08 00:43:25.623018 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.622984    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c489a05b-bf44-4b3c-a2ce-830e78d956fe-kube-api-access-m58zw" (OuterVolumeSpecName: "kube-api-access-m58zw") pod "c489a05b-bf44-4b3c-a2ce-830e78d956fe" (UID: "c489a05b-bf44-4b3c-a2ce-830e78d956fe"). InnerVolumeSpecName "kube-api-access-m58zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:43:25.634261 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.634240    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:25.641517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.641498    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:25.722337 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.722279    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m58zw\" (UniqueName: \"kubernetes.io/projected/c489a05b-bf44-4b3c-a2ce-830e78d956fe-kube-api-access-m58zw\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:43:25.753694 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.753676    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:25.754088 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.754075    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c489a05b-bf44-4b3c-a2ce-830e78d956fe" containerName="test-curl-5884-140284875727792"
Sep 08 00:43:25.754088 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.754089    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c489a05b-bf44-4b3c-a2ce-830e78d956fe" containerName="test-curl-5884-140284875727792"
Sep 08 00:43:25.754181 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.754150    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c489a05b-bf44-4b3c-a2ce-830e78d956fe" containerName="test-curl-5884-140284875727792"
Sep 08 00:43:25.757287 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.757273    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:25.762675 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.762655    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:25.823507 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.823486    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzkxj\" (UniqueName: \"kubernetes.io/projected/52802a76-8ea2-4edd-9b3b-7303d5076eab-kube-api-access-fzkxj\") pod \"test-curl-5884-140284875727792\" (UID: \"52802a76-8ea2-4edd-9b3b-7303d5076eab\") " pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:25.924247 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.924220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzkxj\" (UniqueName: \"kubernetes.io/projected/52802a76-8ea2-4edd-9b3b-7303d5076eab-kube-api-access-fzkxj\") pod \"test-curl-5884-140284875727792\" (UID: \"52802a76-8ea2-4edd-9b3b-7303d5076eab\") " pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:25.931852 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:25.931831    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzkxj\" (UniqueName: \"kubernetes.io/projected/52802a76-8ea2-4edd-9b3b-7303d5076eab-kube-api-access-fzkxj\") pod \"test-curl-5884-140284875727792\" (UID: \"52802a76-8ea2-4edd-9b3b-7303d5076eab\") " pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:26.067647 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.067608    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:26.157606 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.157560    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c489a05b-bf44-4b3c-a2ce-830e78d956fe" path="/var/lib/kubelet/pods/c489a05b-bf44-4b3c-a2ce-830e78d956fe/volumes"
Sep 08 00:43:26.160820 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.160787    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"]
Sep 08 00:43:26.166218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.166196    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.168781 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.168760    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1cf84f-dockercfg-6f2zs\""
Sep 08 00:43:26.168899 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.168783    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1cf84f\""
Sep 08 00:43:26.177338 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.177317    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"]
Sep 08 00:43:26.226661 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.226634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab6739-cb46-4736-ac98-ada1f947a44a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.226752 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.226690    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jpll\" (UniqueName: \"kubernetes.io/projected/b0ab6739-cb46-4736-ac98-ada1f947a44a-kube-api-access-4jpll\") pod \"payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.327141 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.327078    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jpll\" (UniqueName: \"kubernetes.io/projected/b0ab6739-cb46-4736-ac98-ada1f947a44a-kube-api-access-4jpll\") pod \"payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.327265 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.327173    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab6739-cb46-4736-ac98-ada1f947a44a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.327856 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.327833    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab6739-cb46-4736-ac98-ada1f947a44a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.339696 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.339668    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:26.341535 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.341511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jpll\" (UniqueName: \"kubernetes.io/projected/b0ab6739-cb46-4736-ac98-ada1f947a44a-kube-api-access-4jpll\") pod \"payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.342617 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:26.342574    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52802a76_8ea2_4edd_9b3b_7303d5076eab.slice/crio-73e72b7171adcbbc040c3c6724b459fd6491533c0967c22a6ac07cc68b64fed9 WatchSource:0}: Error finding container 73e72b7171adcbbc040c3c6724b459fd6491533c0967c22a6ac07cc68b64fed9: Status 404 returned error can't find the container with id 73e72b7171adcbbc040c3c6724b459fd6491533c0967c22a6ac07cc68b64fed9
Sep 08 00:43:26.445461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.445435    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:26.445564 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.445447    2555 scope.go:117] "RemoveContainer" containerID="15057b91b91e461a6f1ba9559639aecbeb644e2b83819dab65e3a46a698186bb"
Sep 08 00:43:26.446926 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.446900    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5884-140284875727792" event={"ID":"52802a76-8ea2-4edd-9b3b-7303d5076eab","Type":"ContainerStarted","Data":"73e72b7171adcbbc040c3c6724b459fd6491533c0967c22a6ac07cc68b64fed9"}
Sep 08 00:43:26.477768 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.477750    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:26.748967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:26.748940    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"]
Sep 08 00:43:26.751033 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:26.751005    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ab6739_cb46_4736_ac98_ada1f947a44a.slice/crio-a4bbcc3e5b8dfb040e65558924715188a02792ebca35ba4186ea5348ec48698e WatchSource:0}: Error finding container a4bbcc3e5b8dfb040e65558924715188a02792ebca35ba4186ea5348ec48698e: Status 404 returned error can't find the container with id a4bbcc3e5b8dfb040e65558924715188a02792ebca35ba4186ea5348ec48698e
Sep 08 00:43:27.454736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:27.454654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl" event={"ID":"b0ab6739-cb46-4736-ac98-ada1f947a44a","Type":"ContainerStarted","Data":"4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99"}
Sep 08 00:43:27.454736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:27.454713    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl" event={"ID":"b0ab6739-cb46-4736-ac98-ada1f947a44a","Type":"ContainerStarted","Data":"a4bbcc3e5b8dfb040e65558924715188a02792ebca35ba4186ea5348ec48698e"}
Sep 08 00:43:27.454949 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:27.454817    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:27.457311 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:27.457283    2555 generic.go:356] "Generic (PLEG): container finished" podID="52802a76-8ea2-4edd-9b3b-7303d5076eab" containerID="9881d571057c0de8d9a47d2ed35d0dd2717e83c3daf025efad23660dc87dc02c" exitCode=0
Sep 08 00:43:27.457457 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:27.457311    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5884-140284875727792" event={"ID":"52802a76-8ea2-4edd-9b3b-7303d5076eab","Type":"ContainerDied","Data":"9881d571057c0de8d9a47d2ed35d0dd2717e83c3daf025efad23660dc87dc02c"}
Sep 08 00:43:27.472762 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:27.472712    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl" podStartSLOduration=1.472692657 podStartE2EDuration="1.472692657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:27.470423063 +0000 UTC m=+641.813726081" watchObservedRunningTime="2026-09-08 00:43:27.472692657 +0000 UTC m=+641.815995682"
Sep 08 00:43:28.605654 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.605628    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:28.627864 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.627836    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-5884-140284875727792_52802a76-8ea2-4edd-9b3b-7303d5076eab/test-curl-5884-140284875727792/0.log"
Sep 08 00:43:28.649459 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.649426    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:28.656334 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.656309    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-5884-140284875727792"]
Sep 08 00:43:28.739286 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.739198    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:43:28.747950 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.747925    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzkxj\" (UniqueName: \"kubernetes.io/projected/52802a76-8ea2-4edd-9b3b-7303d5076eab-kube-api-access-fzkxj\") pod \"52802a76-8ea2-4edd-9b3b-7303d5076eab\" (UID: \"52802a76-8ea2-4edd-9b3b-7303d5076eab\") "
Sep 08 00:43:28.750704 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.750670    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52802a76-8ea2-4edd-9b3b-7303d5076eab-kube-api-access-fzkxj" (OuterVolumeSpecName: "kube-api-access-fzkxj") pod "52802a76-8ea2-4edd-9b3b-7303d5076eab" (UID: "52802a76-8ea2-4edd-9b3b-7303d5076eab"). InnerVolumeSpecName "kube-api-access-fzkxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:43:28.770674 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.770646    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"]
Sep 08 00:43:28.771243 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.771219    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52802a76-8ea2-4edd-9b3b-7303d5076eab" containerName="test-curl-5884-140284875727792"
Sep 08 00:43:28.771243 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.771244    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="52802a76-8ea2-4edd-9b3b-7303d5076eab" containerName="test-curl-5884-140284875727792"
Sep 08 00:43:28.771493 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.771341    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="52802a76-8ea2-4edd-9b3b-7303d5076eab" containerName="test-curl-5884-140284875727792"
Sep 08 00:43:28.774842 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.774822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.777551 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.777533    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3b0b8262-openshift-default-dockercfg-2h7c9\""
Sep 08 00:43:28.777687 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.777568    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 00:43:28.787531 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.787508    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"]
Sep 08 00:43:28.849569 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-data\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.849751 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-podinfo\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.849751 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849617    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-token\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.849751 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849683    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnnkq\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-kube-api-access-hnnkq\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.849917 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849768    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-socket\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.849917 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-certs\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.850022 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849923    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istiod-ca-cert\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.850022 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.849958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-credential-socket\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.850022 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.850004    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-envoy\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.850152 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.850075    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzkxj\" (UniqueName: \"kubernetes.io/projected/52802a76-8ea2-4edd-9b3b-7303d5076eab-kube-api-access-fzkxj\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:43:28.950537 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-data\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.950717 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-podinfo\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.950717 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950609    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-token\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.950717 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnnkq\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-kube-api-access-hnnkq\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.950717 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-socket\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.950932 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-certs\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.950932 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istiod-ca-cert\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.951037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.950973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-credential-socket\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.951037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.951015    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-envoy\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.951169 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.951144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-data\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.951284 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.951254    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-certs\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.951492 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.951470    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-socket\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.951569 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.951510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-credential-socket\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.951744 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.951715    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istiod-ca-cert\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.953345 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.953328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-podinfo\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.953503 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.953472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-envoy\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.959030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.959003    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnnkq\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-kube-api-access-hnnkq\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:28.959229 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:28.959212    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-token\") pod \"e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") " pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:29.091439 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:29.091395    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:29.403730 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:29.403623    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"]
Sep 08 00:43:29.406746 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:29.406711    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c29a66c_8f4e_4da9_8aa8_28b0e9f86e72.slice/crio-3c420119014b9d2f302718e8278b74fc15e65e927adbe6ab3996f5a87a844e6c WatchSource:0}: Error finding container 3c420119014b9d2f302718e8278b74fc15e65e927adbe6ab3996f5a87a844e6c: Status 404 returned error can't find the container with id 3c420119014b9d2f302718e8278b74fc15e65e927adbe6ab3996f5a87a844e6c
Sep 08 00:43:29.473521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:29.473483    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" event={"ID":"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72","Type":"ContainerStarted","Data":"3c420119014b9d2f302718e8278b74fc15e65e927adbe6ab3996f5a87a844e6c"}
Sep 08 00:43:29.475079 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:29.475051    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:29.475079 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:29.475072    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e72b7171adcbbc040c3c6724b459fd6491533c0967c22a6ac07cc68b64fed9"
Sep 08 00:43:29.485575 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:43:29.485543    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-5884-140284875727792\" is forbidden: User \"system:node:ip-10-0-142-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-98.ec2.internal' and this object" podUID="52802a76-8ea2-4edd-9b3b-7303d5076eab" pod="opendatahub/test-curl-5884-140284875727792"
Sep 08 00:43:30.154346 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:30.154310    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52802a76-8ea2-4edd-9b3b-7303d5076eab" path="/var/lib/kubelet/pods/52802a76-8ea2-4edd-9b3b-7303d5076eab/volumes"
Sep 08 00:43:34.467808 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:34.467766    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 00:43:34.468119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:34.467844    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 00:43:34.468119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:34.467873    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 00:43:35.510349 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:35.510314    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" event={"ID":"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72","Type":"ContainerStarted","Data":"222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af"}
Sep 08 00:43:35.530725 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:35.530676    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" podStartSLOduration=2.472232531 podStartE2EDuration="7.530660147s" totalImagesPullingTime="5.058427616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:35.529008237 +0000 UTC m=+649.872311256" watchObservedRunningTime="2026-09-08 00:43:35.530660147 +0000 UTC m=+649.873963170"
Sep 08 00:43:36.091907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:36.091864    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:36.096690 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:36.096665    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:36.516534 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:36.516445    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:36.517255 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:36.517237    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:38.145106 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.145072    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"]
Sep 08 00:43:38.149372 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.149342    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.152149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.152126    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 00:43:38.153698 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.153504    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 00:43:38.153698 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.153549    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gnd2h\""
Sep 08 00:43:38.153698 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.153555    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3b0b8262-serving-cert\""
Sep 08 00:43:38.153698 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.153642    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 00:43:38.159510 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.159484    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"]
Sep 08 00:43:38.236023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.235989    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwpp\" (UniqueName: \"kubernetes.io/projected/8211f27e-e117-4e9a-b645-166f2096f1bf-kube-api-access-4cwpp\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.236207 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.236162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8211f27e-e117-4e9a-b645-166f2096f1bf-openshift-service-ca\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.236285 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.236212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-maas-api-tls\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.236410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.236389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-metrics-certs\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.337015 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.336982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwpp\" (UniqueName: \"kubernetes.io/projected/8211f27e-e117-4e9a-b645-166f2096f1bf-kube-api-access-4cwpp\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.337160 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.337141    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8211f27e-e117-4e9a-b645-166f2096f1bf-openshift-service-ca\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.337218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.337179    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-maas-api-tls\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.337279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.337217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-metrics-certs\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.337887 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.337865    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8211f27e-e117-4e9a-b645-166f2096f1bf-openshift-service-ca\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.339926 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.339905    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-metrics-certs\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.340008 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.339933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-maas-api-tls\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.348763 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.348741    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwpp\" (UniqueName: \"kubernetes.io/projected/8211f27e-e117-4e9a-b645-166f2096f1bf-kube-api-access-4cwpp\") pod \"maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.464865 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.464778    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:38.467367 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.467342    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:43:38.747813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.747787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"]
Sep 08 00:43:38.749383 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:38.749359    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8211f27e_e117_4e9a_b645_166f2096f1bf.slice/crio-b10cecec80b9388ac59e991a8a97a506160f8fbde9c902a5988cfb762d6c5dc6 WatchSource:0}: Error finding container b10cecec80b9388ac59e991a8a97a506160f8fbde9c902a5988cfb762d6c5dc6: Status 404 returned error can't find the container with id b10cecec80b9388ac59e991a8a97a506160f8fbde9c902a5988cfb762d6c5dc6
Sep 08 00:43:38.837162 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.837039    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"]
Sep 08 00:43:38.842841 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.842822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:38.845353 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.845326    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3b0b8262\""
Sep 08 00:43:38.845464 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.845361    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3b0b8262-dockercfg-4r2g4\""
Sep 08 00:43:38.850303 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.850280    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"]
Sep 08 00:43:38.943153 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.943119    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88edd372-b340-429f-8095-53f3c308022b-plugins-config-volume\") pod \"payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") " pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:38.943313 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:38.943164    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjrn\" (UniqueName: \"kubernetes.io/projected/88edd372-b340-429f-8095-53f3c308022b-kube-api-access-nxjrn\") pod \"payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") " pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:39.043914 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.043840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88edd372-b340-429f-8095-53f3c308022b-plugins-config-volume\") pod \"payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") " pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:39.043914 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.043889    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjrn\" (UniqueName: \"kubernetes.io/projected/88edd372-b340-429f-8095-53f3c308022b-kube-api-access-nxjrn\") pod \"payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") " pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:39.044527 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.044507    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88edd372-b340-429f-8095-53f3c308022b-plugins-config-volume\") pod \"payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") " pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:39.051770 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.051746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjrn\" (UniqueName: \"kubernetes.io/projected/88edd372-b340-429f-8095-53f3c308022b-kube-api-access-nxjrn\") pod \"payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") " pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:39.155287 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.155259    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:39.430378 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.430342    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"]
Sep 08 00:43:39.433095 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:39.433064    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88edd372_b340_429f_8095_53f3c308022b.slice/crio-984d50f88ce45b8affab3efdb07c8087a65f4a4a89225ee09d6aa0fa1637f6b6 WatchSource:0}: Error finding container 984d50f88ce45b8affab3efdb07c8087a65f4a4a89225ee09d6aa0fa1637f6b6: Status 404 returned error can't find the container with id 984d50f88ce45b8affab3efdb07c8087a65f4a4a89225ee09d6aa0fa1637f6b6
Sep 08 00:43:39.539243 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.539198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp" event={"ID":"8211f27e-e117-4e9a-b645-166f2096f1bf","Type":"ContainerStarted","Data":"b10cecec80b9388ac59e991a8a97a506160f8fbde9c902a5988cfb762d6c5dc6"}
Sep 08 00:43:39.541131 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.541070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk" event={"ID":"88edd372-b340-429f-8095-53f3c308022b","Type":"ContainerStarted","Data":"fdf98e7908b35630f4952fd73b043c9a418b170ad8f77851e9be707ed478f328"}
Sep 08 00:43:39.541131 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.541119    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk" event={"ID":"88edd372-b340-429f-8095-53f3c308022b","Type":"ContainerStarted","Data":"984d50f88ce45b8affab3efdb07c8087a65f4a4a89225ee09d6aa0fa1637f6b6"}
Sep 08 00:43:39.541312 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.541226    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:39.558160 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:39.558111    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk" podStartSLOduration=1.558093225 podStartE2EDuration="1.558093225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:39.555723181 +0000 UTC m=+653.899026227" watchObservedRunningTime="2026-09-08 00:43:39.558093225 +0000 UTC m=+653.901396249"
Sep 08 00:43:40.549047 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:40.548919    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp" event={"ID":"8211f27e-e117-4e9a-b645-166f2096f1bf","Type":"ContainerStarted","Data":"b6b160795575d5075dd64edbdce8938d3547dc08acbaaee2aa362ac8968f3869"}
Sep 08 00:43:40.568180 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:40.568135    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp" podStartSLOduration=1.1311189050000001 podStartE2EDuration="2.568119494s" totalImagesPullingTime="1.437000589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:40.566024353 +0000 UTC m=+654.909327383" watchObservedRunningTime="2026-09-08 00:43:40.568119494 +0000 UTC m=+654.911422540"
Sep 08 00:43:41.553850 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:41.553820    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:47.564013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:47.563980    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:43:50.550744 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:50.550713    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:43:55.652896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:55.652859    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerID="222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af" exitCode=0
Sep 08 00:43:55.653386 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:55.652934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" event={"ID":"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72","Type":"ContainerDied","Data":"222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af"}
Sep 08 00:43:55.653386 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:55.653348    2555 scope.go:117] "RemoveContainer" containerID="222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af"
Sep 08 00:43:55.654729 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:55.654696    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 00:43:55.654817 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:55.654766    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 00:43:55.654856 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:55.654811    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 00:43:56.660572 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:56.660534    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" event={"ID":"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72","Type":"ContainerStarted","Data":"e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec"}
Sep 08 00:43:57.092343 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.092306    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:43:57.092692 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.092361    2555 patch_prober.go:28] interesting pod/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 08 00:43:57.092890 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.092863    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 08 00:43:57.496285 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.496183    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"]
Sep 08 00:43:57.500064 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.500035    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.502539 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.502511    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e3896b-serving-cert\""
Sep 08 00:43:57.510512 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.510491    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"]
Sep 08 00:43:57.629368 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.629334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-metrics-certs\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.629541 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.629444    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.629541 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.629473    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbtn\" (UniqueName: \"kubernetes.io/projected/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-kube-api-access-hbbtn\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.629541 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.629507    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.730118 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.730086    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.730478 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.730132    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbtn\" (UniqueName: \"kubernetes.io/projected/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-kube-api-access-hbbtn\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.730478 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.730170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.730478 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.730276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-metrics-certs\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.730863 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.730837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.732723 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.732702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-metrics-certs\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.732795 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.732775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-maas-api-tls\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.738297 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.738271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbtn\" (UniqueName: \"kubernetes.io/projected/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-kube-api-access-hbbtn\") pod \"maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:57.811295 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:57.811263    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:58.089563 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:58.089473    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"]
Sep 08 00:43:58.101815 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:58.091972    2555 patch_prober.go:28] interesting pod/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 08 00:43:58.101815 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:58.092127    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 08 00:43:58.103716 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:43:58.103668    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0400d54a_e6e1_483e_ab9a_7a2abfdce0ec.slice/crio-71f3f3829c6d47104d2abc91b6a6192ddf8b7c30af43337c40bd190391e1bbe8 WatchSource:0}: Error finding container 71f3f3829c6d47104d2abc91b6a6192ddf8b7c30af43337c40bd190391e1bbe8: Status 404 returned error can't find the container with id 71f3f3829c6d47104d2abc91b6a6192ddf8b7c30af43337c40bd190391e1bbe8
Sep 08 00:43:58.673864 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:58.673818    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr" event={"ID":"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec","Type":"ContainerStarted","Data":"8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b"}
Sep 08 00:43:58.674043 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:58.673890    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr" event={"ID":"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec","Type":"ContainerStarted","Data":"71f3f3829c6d47104d2abc91b6a6192ddf8b7c30af43337c40bd190391e1bbe8"}
Sep 08 00:43:58.674733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:58.674704    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:43:59.092744 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:59.092702    2555 patch_prober.go:28] interesting pod/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 08 00:43:59.093234 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:59.092767    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 08 00:43:59.832021 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:59.831969    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr" podStartSLOduration=2.432286176 podStartE2EDuration="2.831953358s" totalImagesPullingTime="399.667182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:58.696057086 +0000 UTC m=+673.039360107" watchObservedRunningTime="2026-09-08 00:43:59.831953358 +0000 UTC m=+674.175256377"
Sep 08 00:43:59.833983 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:43:59.833963    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:44:00.092655 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:00.092541    2555 patch_prober.go:28] interesting pod/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 08 00:44:00.092655 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:00.092628    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 08 00:44:01.092317 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:01.092281    2555 patch_prober.go:28] interesting pod/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 08 00:44:01.092584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:01.092556    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 08 00:44:02.096039 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:02.096009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:44:02.096419 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:02.096379    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:44:02.097044 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:02.097025    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:44:05.696927 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:05.696895    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:44:18.566678 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.566637    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"]
Sep 08 00:44:18.567092 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.566873    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp" podUID="8211f27e-e117-4e9a-b645-166f2096f1bf" containerName="maas-api" containerID="cri-o://b6b160795575d5075dd64edbdce8938d3547dc08acbaaee2aa362ac8968f3869" gracePeriod=30
Sep 08 00:44:18.622485 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.622446    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"]
Sep 08 00:44:18.623081 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.622759    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk" podUID="88edd372-b340-429f-8095-53f3c308022b" containerName="payload-processing" containerID="cri-o://fdf98e7908b35630f4952fd73b043c9a418b170ad8f77851e9be707ed478f328" gracePeriod=30
Sep 08 00:44:18.797988 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.797953    2555 generic.go:356] "Generic (PLEG): container finished" podID="8211f27e-e117-4e9a-b645-166f2096f1bf" containerID="b6b160795575d5075dd64edbdce8938d3547dc08acbaaee2aa362ac8968f3869" exitCode=0
Sep 08 00:44:18.798174 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.798038    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp" event={"ID":"8211f27e-e117-4e9a-b645-166f2096f1bf","Type":"ContainerDied","Data":"b6b160795575d5075dd64edbdce8938d3547dc08acbaaee2aa362ac8968f3869"}
Sep 08 00:44:18.799715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.799688    2555 generic.go:356] "Generic (PLEG): container finished" podID="88edd372-b340-429f-8095-53f3c308022b" containerID="fdf98e7908b35630f4952fd73b043c9a418b170ad8f77851e9be707ed478f328" exitCode=0
Sep 08 00:44:18.799876 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.799735    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk" event={"ID":"88edd372-b340-429f-8095-53f3c308022b","Type":"ContainerDied","Data":"fdf98e7908b35630f4952fd73b043c9a418b170ad8f77851e9be707ed478f328"}
Sep 08 00:44:18.877858 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.877824    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:44:18.919840 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.919795    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:44:18.944891 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.944858    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"]
Sep 08 00:44:18.945144 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:18.945120    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy" containerID="cri-o://e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec" gracePeriod=30
Sep 08 00:44:19.024069 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024038    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88edd372-b340-429f-8095-53f3c308022b-plugins-config-volume\") pod \"88edd372-b340-429f-8095-53f3c308022b\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") "
Sep 08 00:44:19.024238 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024075    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-metrics-certs\") pod \"8211f27e-e117-4e9a-b645-166f2096f1bf\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") "
Sep 08 00:44:19.024238 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024095    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxjrn\" (UniqueName: \"kubernetes.io/projected/88edd372-b340-429f-8095-53f3c308022b-kube-api-access-nxjrn\") pod \"88edd372-b340-429f-8095-53f3c308022b\" (UID: \"88edd372-b340-429f-8095-53f3c308022b\") "
Sep 08 00:44:19.024238 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024117    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cwpp\" (UniqueName: \"kubernetes.io/projected/8211f27e-e117-4e9a-b645-166f2096f1bf-kube-api-access-4cwpp\") pod \"8211f27e-e117-4e9a-b645-166f2096f1bf\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") "
Sep 08 00:44:19.024238 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024179    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-maas-api-tls\") pod \"8211f27e-e117-4e9a-b645-166f2096f1bf\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") "
Sep 08 00:44:19.024481 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024252    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8211f27e-e117-4e9a-b645-166f2096f1bf-openshift-service-ca\") pod \"8211f27e-e117-4e9a-b645-166f2096f1bf\" (UID: \"8211f27e-e117-4e9a-b645-166f2096f1bf\") "
Sep 08 00:44:19.024558 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024520    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88edd372-b340-429f-8095-53f3c308022b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88edd372-b340-429f-8095-53f3c308022b" (UID: "88edd372-b340-429f-8095-53f3c308022b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:44:19.024727 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.024702    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8211f27e-e117-4e9a-b645-166f2096f1bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8211f27e-e117-4e9a-b645-166f2096f1bf" (UID: "8211f27e-e117-4e9a-b645-166f2096f1bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:44:19.026349 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.026322    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8211f27e-e117-4e9a-b645-166f2096f1bf-kube-api-access-4cwpp" (OuterVolumeSpecName: "kube-api-access-4cwpp") pod "8211f27e-e117-4e9a-b645-166f2096f1bf" (UID: "8211f27e-e117-4e9a-b645-166f2096f1bf"). InnerVolumeSpecName "kube-api-access-4cwpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:44:19.026454 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.026345    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88edd372-b340-429f-8095-53f3c308022b-kube-api-access-nxjrn" (OuterVolumeSpecName: "kube-api-access-nxjrn") pod "88edd372-b340-429f-8095-53f3c308022b" (UID: "88edd372-b340-429f-8095-53f3c308022b"). InnerVolumeSpecName "kube-api-access-nxjrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:44:19.026627 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.026605    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8211f27e-e117-4e9a-b645-166f2096f1bf" (UID: "8211f27e-e117-4e9a-b645-166f2096f1bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:44:19.026714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.026626    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8211f27e-e117-4e9a-b645-166f2096f1bf" (UID: "8211f27e-e117-4e9a-b645-166f2096f1bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:44:19.125790 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.125715    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8211f27e-e117-4e9a-b645-166f2096f1bf-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:19.125790 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.125739    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88edd372-b340-429f-8095-53f3c308022b-plugins-config-volume\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:19.125790 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.125750    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:19.125790 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.125759    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxjrn\" (UniqueName: \"kubernetes.io/projected/88edd372-b340-429f-8095-53f3c308022b-kube-api-access-nxjrn\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:19.125790 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.125767    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwpp\" (UniqueName: \"kubernetes.io/projected/8211f27e-e117-4e9a-b645-166f2096f1bf-kube-api-access-4cwpp\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:19.125790 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.125775    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8211f27e-e117-4e9a-b645-166f2096f1bf-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:19.806957 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.806928    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"
Sep 08 00:44:19.807345 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.806923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp" event={"ID":"8211f27e-e117-4e9a-b645-166f2096f1bf","Type":"ContainerDied","Data":"b10cecec80b9388ac59e991a8a97a506160f8fbde9c902a5988cfb762d6c5dc6"}
Sep 08 00:44:19.807345 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.807078    2555 scope.go:117] "RemoveContainer" containerID="b6b160795575d5075dd64edbdce8938d3547dc08acbaaee2aa362ac8968f3869"
Sep 08 00:44:19.809159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.809128    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk" event={"ID":"88edd372-b340-429f-8095-53f3c308022b","Type":"ContainerDied","Data":"984d50f88ce45b8affab3efdb07c8087a65f4a4a89225ee09d6aa0fa1637f6b6"}
Sep 08 00:44:19.809268 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.809188    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"
Sep 08 00:44:19.819329 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.819241    2555 scope.go:117] "RemoveContainer" containerID="fdf98e7908b35630f4952fd73b043c9a418b170ad8f77851e9be707ed478f328"
Sep 08 00:44:19.835004 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.834954    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"]
Sep 08 00:44:19.845341 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.845320    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0b8262-575f9464bd-bb7cp"]
Sep 08 00:44:19.860971 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.860911    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"]
Sep 08 00:44:19.863818 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:19.863792    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b0b8262-5c8cdd88d5-s4vgk"]
Sep 08 00:44:20.153929 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:20.153837    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8211f27e-e117-4e9a-b645-166f2096f1bf" path="/var/lib/kubelet/pods/8211f27e-e117-4e9a-b645-166f2096f1bf/volumes"
Sep 08 00:44:20.154298 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:20.154276    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88edd372-b340-429f-8095-53f3c308022b" path="/var/lib/kubelet/pods/88edd372-b340-429f-8095-53f3c308022b/volumes"
Sep 08 00:44:22.272491 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.272459    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"]
Sep 08 00:44:22.272978 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.272961    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8211f27e-e117-4e9a-b645-166f2096f1bf" containerName="maas-api"
Sep 08 00:44:22.273031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.272980    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8211f27e-e117-4e9a-b645-166f2096f1bf" containerName="maas-api"
Sep 08 00:44:22.273031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.272996    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88edd372-b340-429f-8095-53f3c308022b" containerName="payload-processing"
Sep 08 00:44:22.273031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.273002    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="88edd372-b340-429f-8095-53f3c308022b" containerName="payload-processing"
Sep 08 00:44:22.273138 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.273072    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8211f27e-e117-4e9a-b645-166f2096f1bf" containerName="maas-api"
Sep 08 00:44:22.273138 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.273083    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="88edd372-b340-429f-8095-53f3c308022b" containerName="payload-processing"
Sep 08 00:44:22.275951 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.275926    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.278538 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.278519    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3790db3d-serving-cert\""
Sep 08 00:44:22.292970 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.292943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"]
Sep 08 00:44:22.355704 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.355679    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-metrics-certs\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.355824 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.355709    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ll2\" (UniqueName: \"kubernetes.io/projected/6468ea6b-c870-4904-8484-317716559581-kube-api-access-f4ll2\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.355824 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.355736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.355900 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.355843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468ea6b-c870-4904-8484-317716559581-openshift-service-ca\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.457218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.457186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-metrics-certs\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.457218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.457218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ll2\" (UniqueName: \"kubernetes.io/projected/6468ea6b-c870-4904-8484-317716559581-kube-api-access-f4ll2\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.457404 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.457245    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.457404 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.457294    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468ea6b-c870-4904-8484-317716559581-openshift-service-ca\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.457404 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:44:22.457394    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-serving-cert: secret "maas-api-e2e-migrate-3790db3d-serving-cert" not found
Sep 08 00:44:22.457538 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:44:22.457464    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls podName:6468ea6b-c870-4904-8484-317716559581 nodeName:}" failed. No retries permitted until 2026-09-08 00:44:22.957443879 +0000 UTC m=+697.300746878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls") pod "maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p" (UID: "6468ea6b-c870-4904-8484-317716559581") : secret "maas-api-e2e-migrate-3790db3d-serving-cert" not found
Sep 08 00:44:22.457952 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.457920    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468ea6b-c870-4904-8484-317716559581-openshift-service-ca\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.459535 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.459515    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-metrics-certs\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.465130 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.465112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ll2\" (UniqueName: \"kubernetes.io/projected/6468ea6b-c870-4904-8484-317716559581-kube-api-access-f4ll2\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.961919 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.961885    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:22.964279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:22.964257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls\") pod \"maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:23.189355 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:23.189316    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:23.459838 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:23.459809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"]
Sep 08 00:44:23.461241 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:44:23.461204    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6468ea6b_c870_4904_8484_317716559581.slice/crio-790711dfe15e97f26ed45aaf79425ee2a41a3d4a91c40b8ef4c76089ea9299e6 WatchSource:0}: Error finding container 790711dfe15e97f26ed45aaf79425ee2a41a3d4a91c40b8ef4c76089ea9299e6: Status 404 returned error can't find the container with id 790711dfe15e97f26ed45aaf79425ee2a41a3d4a91c40b8ef4c76089ea9299e6
Sep 08 00:44:23.838642 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:23.838584    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p" event={"ID":"6468ea6b-c870-4904-8484-317716559581","Type":"ContainerStarted","Data":"790711dfe15e97f26ed45aaf79425ee2a41a3d4a91c40b8ef4c76089ea9299e6"}
Sep 08 00:44:24.155887 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.155854    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:44:24.209532 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.209502    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:44:24.374238 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374204    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istiod-ca-cert\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374397 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374254    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-envoy\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374397 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374318    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-data\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374397 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374363    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnnkq\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-kube-api-access-hnnkq\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374449    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-podinfo\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374501    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-token\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374693 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374659    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-credential-socket\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374752 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374713    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-certs\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374807 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374756    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:44:24.374807 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374778    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-socket\") pod \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\" (UID: \"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72\") "
Sep 08 00:44:24.374940 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.374912    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:44:24.375049 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375029    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:44:24.375119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375043    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:44:24.375119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375025    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:44:24.375237 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375219    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-credential-socket\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.375292 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375240    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.375292 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375255    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-workload-socket\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.375292 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375268    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istiod-ca-cert\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.375292 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.375281    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-data\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.377344 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.377313    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:44:24.377464 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.377369    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-kube-api-access-hnnkq" (OuterVolumeSpecName: "kube-api-access-hnnkq") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "kube-api-access-hnnkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:44:24.377464 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.377396    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 00:44:24.377464 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.377448    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" (UID: "3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:44:24.475732 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.475707    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-envoy\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.476043 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.475733    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnnkq\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-kube-api-access-hnnkq\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.476043 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.475747    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-podinfo\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.476043 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.475759    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72-istio-token\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:24.845891 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.845858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p" event={"ID":"6468ea6b-c870-4904-8484-317716559581","Type":"ContainerStarted","Data":"84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5"}
Sep 08 00:44:24.846083 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.845956    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:24.847761 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.847736    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerID="e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec" exitCode=0
Sep 08 00:44:24.847879 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.847769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" event={"ID":"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72","Type":"ContainerDied","Data":"e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec"}
Sep 08 00:44:24.847879 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.847774    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"
Sep 08 00:44:24.847879 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.847797    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg" event={"ID":"3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72","Type":"ContainerDied","Data":"3c420119014b9d2f302718e8278b74fc15e65e927adbe6ab3996f5a87a844e6c"}
Sep 08 00:44:24.847879 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.847820    2555 scope.go:117] "RemoveContainer" containerID="e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec"
Sep 08 00:44:24.858312 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.858297    2555 scope.go:117] "RemoveContainer" containerID="222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af"
Sep 08 00:44:24.864646 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.864584    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p" podStartSLOduration=2.370051823 podStartE2EDuration="2.864567627s" totalImagesPullingTime="494.515804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:44:24.864529865 +0000 UTC m=+699.207832886" watchObservedRunningTime="2026-09-08 00:44:24.864567627 +0000 UTC m=+699.207870643"
Sep 08 00:44:24.869908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.869887    2555 scope.go:117] "RemoveContainer" containerID="e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec"
Sep 08 00:44:24.870151 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:44:24.870132    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec\": container with ID starting with e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec not found: ID does not exist" containerID="e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec"
Sep 08 00:44:24.870208 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.870158    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec"} err="failed to get container status \"e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec\": rpc error: code = NotFound desc = could not find container \"e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec\": container with ID starting with e54638aec6888116ade82b579cb6e7ca78f20dde3c26888a9f957e2faa030fec not found: ID does not exist"
Sep 08 00:44:24.870208 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.870175    2555 scope.go:117] "RemoveContainer" containerID="222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af"
Sep 08 00:44:24.870409 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:44:24.870393    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af\": container with ID starting with 222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af not found: ID does not exist" containerID="222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af"
Sep 08 00:44:24.870451 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.870415    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af"} err="failed to get container status \"222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af\": rpc error: code = NotFound desc = could not find container \"222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af\": container with ID starting with 222e1028721cca7c1736c2fc6d9b540fed047a9840d23436bb43ba305ab534af not found: ID does not exist"
Sep 08 00:44:24.879289 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.879261    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"]
Sep 08 00:44:24.882440 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:24.882415    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3b0b8262-openshift-default-6985bfcd4b-q6xrg"]
Sep 08 00:44:26.153394 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:26.153362    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" path="/var/lib/kubelet/pods/3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72/volumes"
Sep 08 00:44:30.861255 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:30.861224    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:53.567922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.567888    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"]
Sep 08 00:44:53.568384 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.568104    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p" podUID="6468ea6b-c870-4904-8484-317716559581" containerName="maas-api" containerID="cri-o://84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5" gracePeriod=30
Sep 08 00:44:53.823668 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.823604    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:53.916764 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.916735    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-metrics-certs\") pod \"6468ea6b-c870-4904-8484-317716559581\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") "
Sep 08 00:44:53.916967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.916800    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls\") pod \"6468ea6b-c870-4904-8484-317716559581\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") "
Sep 08 00:44:53.916967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.916859    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4ll2\" (UniqueName: \"kubernetes.io/projected/6468ea6b-c870-4904-8484-317716559581-kube-api-access-f4ll2\") pod \"6468ea6b-c870-4904-8484-317716559581\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") "
Sep 08 00:44:53.916967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.916892    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468ea6b-c870-4904-8484-317716559581-openshift-service-ca\") pod \"6468ea6b-c870-4904-8484-317716559581\" (UID: \"6468ea6b-c870-4904-8484-317716559581\") "
Sep 08 00:44:53.917309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.917283    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6468ea6b-c870-4904-8484-317716559581-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6468ea6b-c870-4904-8484-317716559581" (UID: "6468ea6b-c870-4904-8484-317716559581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:44:53.919323 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.919157    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6468ea6b-c870-4904-8484-317716559581" (UID: "6468ea6b-c870-4904-8484-317716559581"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:44:53.919323 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.919203    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6468ea6b-c870-4904-8484-317716559581-kube-api-access-f4ll2" (OuterVolumeSpecName: "kube-api-access-f4ll2") pod "6468ea6b-c870-4904-8484-317716559581" (UID: "6468ea6b-c870-4904-8484-317716559581"). InnerVolumeSpecName "kube-api-access-f4ll2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:44:53.919323 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:53.919210    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6468ea6b-c870-4904-8484-317716559581" (UID: "6468ea6b-c870-4904-8484-317716559581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:44:54.005290 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.005250    2555 generic.go:356] "Generic (PLEG): container finished" podID="6468ea6b-c870-4904-8484-317716559581" containerID="84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5" exitCode=0
Sep 08 00:44:54.005468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.005319    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"
Sep 08 00:44:54.005468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.005342    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p" event={"ID":"6468ea6b-c870-4904-8484-317716559581","Type":"ContainerDied","Data":"84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5"}
Sep 08 00:44:54.005468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.005400    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p" event={"ID":"6468ea6b-c870-4904-8484-317716559581","Type":"ContainerDied","Data":"790711dfe15e97f26ed45aaf79425ee2a41a3d4a91c40b8ef4c76089ea9299e6"}
Sep 08 00:44:54.005468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.005425    2555 scope.go:117] "RemoveContainer" containerID="84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5"
Sep 08 00:44:54.015282 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.015261    2555 scope.go:117] "RemoveContainer" containerID="84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5"
Sep 08 00:44:54.015551 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:44:54.015532    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5\": container with ID starting with 84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5 not found: ID does not exist" containerID="84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5"
Sep 08 00:44:54.015630 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.015559    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5"} err="failed to get container status \"84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5\": rpc error: code = NotFound desc = could not find container \"84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5\": container with ID starting with 84eee91386fe1bb0ab21100903911f56b0b83e140c4bf097fb223f050aceeee5 not found: ID does not exist"
Sep 08 00:44:54.017827 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.017810    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:54.017904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.017829    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4ll2\" (UniqueName: \"kubernetes.io/projected/6468ea6b-c870-4904-8484-317716559581-kube-api-access-f4ll2\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:54.017904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.017841    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468ea6b-c870-4904-8484-317716559581-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:54.017904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.017850    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468ea6b-c870-4904-8484-317716559581-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:44:54.027210 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.027190    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"]
Sep 08 00:44:54.033636 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.029522    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3790db3d-67b598db46-kfm7p"]
Sep 08 00:44:54.153468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.153385    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6468ea6b-c870-4904-8484-317716559581" path="/var/lib/kubelet/pods/6468ea6b-c870-4904-8484-317716559581/volumes"
Sep 08 00:44:54.646530 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:54.646498    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:44:58.555276 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.555236    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"]
Sep 08 00:44:58.555887 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.555865    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy"
Sep 08 00:44:58.555976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.555890    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy"
Sep 08 00:44:58.555976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.555937    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6468ea6b-c870-4904-8484-317716559581" containerName="maas-api"
Sep 08 00:44:58.555976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.555946    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6468ea6b-c870-4904-8484-317716559581" containerName="maas-api"
Sep 08 00:44:58.556121 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.556039    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy"
Sep 08 00:44:58.556121 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.556055    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6468ea6b-c870-4904-8484-317716559581" containerName="maas-api"
Sep 08 00:44:58.560810 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.560788    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.563402 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.563381    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f80520f6-serving-cert\""
Sep 08 00:44:58.572182 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.572149    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"]
Sep 08 00:44:58.653862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.653828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-metrics-certs\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.654000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.653884    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-openshift-service-ca\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.654000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.653935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-maas-api-tls\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.654000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.653981    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6ng\" (UniqueName: \"kubernetes.io/projected/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-kube-api-access-vs6ng\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.755468 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.755438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-openshift-service-ca\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.755635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.755472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-maas-api-tls\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.755635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.755506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6ng\" (UniqueName: \"kubernetes.io/projected/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-kube-api-access-vs6ng\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.755635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.755588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-metrics-certs\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.756189 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.756168    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-openshift-service-ca\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.757833 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.757814    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-metrics-certs\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.757950 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.757930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-maas-api-tls\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.763625 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.763603    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6ng\" (UniqueName: \"kubernetes.io/projected/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-kube-api-access-vs6ng\") pod \"maas-api-e2e-ait-f80520f6-7689c75866-dzpwx\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:58.873108 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:58.873046    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:44:59.149949 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:44:59.149918    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"]
Sep 08 00:44:59.152784 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:44:59.152752    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34591cf7_32d0_4ee6_a9b8_64de2fb85f84.slice/crio-7be60f90ccd6600e9b9d62c144a152fe86d566528e2316f24ca0d06201faa7c8 WatchSource:0}: Error finding container 7be60f90ccd6600e9b9d62c144a152fe86d566528e2316f24ca0d06201faa7c8: Status 404 returned error can't find the container with id 7be60f90ccd6600e9b9d62c144a152fe86d566528e2316f24ca0d06201faa7c8
Sep 08 00:45:00.041395 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.041311    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx" event={"ID":"34591cf7-32d0-4ee6-a9b8-64de2fb85f84","Type":"ContainerStarted","Data":"881e883050b033f3bc96ec52017d480422824d23fe2eeed5a86cfa5863c7826b"}
Sep 08 00:45:00.041395 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.041361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx" event={"ID":"34591cf7-32d0-4ee6-a9b8-64de2fb85f84","Type":"ContainerStarted","Data":"7be60f90ccd6600e9b9d62c144a152fe86d566528e2316f24ca0d06201faa7c8"}
Sep 08 00:45:00.041835 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.041446    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:45:00.058231 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.058178    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx" podStartSLOduration=1.470218515 podStartE2EDuration="2.058164587s" totalImagesPullingTime="587.946072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:00.055927784 +0000 UTC m=+734.399230804" watchObservedRunningTime="2026-09-08 00:45:00.058164587 +0000 UTC m=+734.401467605"
Sep 08 00:45:00.142287 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.142253    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"]
Sep 08 00:45:00.142974 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.142887    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy"
Sep 08 00:45:00.142974 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.142909    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy"
Sep 08 00:45:00.143180 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.143028    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c29a66c-8f4e-4da9-8aa8-28b0e9f86e72" containerName="istio-proxy"
Sep 08 00:45:00.146337 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.146316    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"]
Sep 08 00:45:00.146457 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.146426    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"
Sep 08 00:45:00.148918 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.148878    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\""
Sep 08 00:45:00.150753 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.150730    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"
Sep 08 00:45:00.162352 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.162327    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"]
Sep 08 00:45:00.162449 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.162377    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"]
Sep 08 00:45:00.162449 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.162397    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"
Sep 08 00:45:00.171034 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.171012    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"
Sep 08 00:45:00.449662 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.449505    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"]
Sep 08 00:45:00.450968 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:45:00.450940    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a72775_eb35_4e5d_a1e9_8d8f87c3ac1d.slice/crio-14204c0b87d89e43418743285a1a71ed35edd18d38182fc828df881eb50e3db7 WatchSource:0}: Error finding container 14204c0b87d89e43418743285a1a71ed35edd18d38182fc828df881eb50e3db7: Status 404 returned error can't find the container with id 14204c0b87d89e43418743285a1a71ed35edd18d38182fc828df881eb50e3db7
Sep 08 00:45:00.451714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:00.451686    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"]
Sep 08 00:45:00.453911 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:45:00.453891    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb765c3_9b95_4901_92d2_dcb6bdd8127c.slice/crio-3697c54fc4810d84c6247b795e3913e4f2a992461eb40b3448ba0e65f73f3d42 WatchSource:0}: Error finding container 3697c54fc4810d84c6247b795e3913e4f2a992461eb40b3448ba0e65f73f3d42: Status 404 returned error can't find the container with id 3697c54fc4810d84c6247b795e3913e4f2a992461eb40b3448ba0e65f73f3d42
Sep 08 00:45:01.048802 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:01.048766    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b" event={"ID":"f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d","Type":"ContainerStarted","Data":"14204c0b87d89e43418743285a1a71ed35edd18d38182fc828df881eb50e3db7"}
Sep 08 00:45:01.050019 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:01.049991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm" event={"ID":"9cb765c3-9b95-4901-92d2-dcb6bdd8127c","Type":"ContainerStarted","Data":"3697c54fc4810d84c6247b795e3913e4f2a992461eb40b3448ba0e65f73f3d42"}
Sep 08 00:45:04.070967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:04.070937    2555 generic.go:356] "Generic (PLEG): container finished" podID="f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d" containerID="43930da827a2bf54665ffff491d94bff1599d00734b13b914af44a3c3db2ba37" exitCode=0
Sep 08 00:45:04.071310 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:04.071023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b" event={"ID":"f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d","Type":"ContainerDied","Data":"43930da827a2bf54665ffff491d94bff1599d00734b13b914af44a3c3db2ba37"}
Sep 08 00:45:04.072696 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:04.072671    2555 generic.go:356] "Generic (PLEG): container finished" podID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerID="58f28fac4aa79376582ed40a3c1ffef176f622b13657c981ddb92017ceec341d" exitCode=7
Sep 08 00:45:04.072803 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:04.072755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm" event={"ID":"9cb765c3-9b95-4901-92d2-dcb6bdd8127c","Type":"ContainerDied","Data":"58f28fac4aa79376582ed40a3c1ffef176f622b13657c981ddb92017ceec341d"}
Sep 08 00:45:04.072885 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:04.072871    2555 scope.go:117] "RemoveContainer" containerID="58f28fac4aa79376582ed40a3c1ffef176f622b13657c981ddb92017ceec341d"
Sep 08 00:45:05.082458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:05.082423    2555 generic.go:356] "Generic (PLEG): container finished" podID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerID="406b1c255c7e34892afb175eb05aca0387aab210801f4bc45dc2fe666e060888" exitCode=7
Sep 08 00:45:05.082873 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:05.082484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm" event={"ID":"9cb765c3-9b95-4901-92d2-dcb6bdd8127c","Type":"ContainerDied","Data":"406b1c255c7e34892afb175eb05aca0387aab210801f4bc45dc2fe666e060888"}
Sep 08 00:45:05.082873 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:05.082545    2555 scope.go:117] "RemoveContainer" containerID="58f28fac4aa79376582ed40a3c1ffef176f622b13657c981ddb92017ceec341d"
Sep 08 00:45:05.082873 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:05.082772    2555 scope.go:117] "RemoveContainer" containerID="406b1c255c7e34892afb175eb05aca0387aab210801f4bc45dc2fe666e060888"
Sep 08 00:45:05.083382 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:45:05.083030    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm_odh-ai-gateway-infra(9cb765c3-9b95-4901-92d2-dcb6bdd8127c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c"
Sep 08 00:45:05.230515 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:05.230492    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"
Sep 08 00:45:06.055461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:06.055434    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:45:06.090921 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:06.090890    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"
Sep 08 00:45:06.090921 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:06.090911    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b" event={"ID":"f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d","Type":"ContainerDied","Data":"14204c0b87d89e43418743285a1a71ed35edd18d38182fc828df881eb50e3db7"}
Sep 08 00:45:06.091406 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:06.090959    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14204c0b87d89e43418743285a1a71ed35edd18d38182fc828df881eb50e3db7"
Sep 08 00:45:06.093209 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:06.093190    2555 scope.go:117] "RemoveContainer" containerID="406b1c255c7e34892afb175eb05aca0387aab210801f4bc45dc2fe666e060888"
Sep 08 00:45:06.093427 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:45:06.093397    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm_odh-ai-gateway-infra(9cb765c3-9b95-4901-92d2-dcb6bdd8127c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c"
Sep 08 00:45:16.420181 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.420142    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"]
Sep 08 00:45:16.420659 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.420547    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d" containerName="cleanup"
Sep 08 00:45:16.420659 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.420557    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d" containerName="cleanup"
Sep 08 00:45:16.420659 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.420653    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d" containerName="cleanup"
Sep 08 00:45:16.423731 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.423711    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.426371 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.426352    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-31f424\"/\"default-dockercfg-dzzw8\""
Sep 08 00:45:16.427821 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.427795    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-31f424\"/\"kube-root-ca.crt\""
Sep 08 00:45:16.427936 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.427854    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-31f424\"/\"openshift-service-ca.crt\""
Sep 08 00:45:16.428117 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.428096    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-31f424\"/\"autores-model-31f424-kserve-self-signed-certs\""
Sep 08 00:45:16.430654 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.430633    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"]
Sep 08 00:45:16.511543 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.511506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-home\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.511734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.511562    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-dshm\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.511734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.511625    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b1673-4da0-4b3d-aadb-ffacccff92c2-tls-certs\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.511734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.511682    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-model-cache\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.511734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.511714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wd4\" (UniqueName: \"kubernetes.io/projected/269b1673-4da0-4b3d-aadb-ffacccff92c2-kube-api-access-d8wd4\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.511734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.511734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-tmp-dir\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.612698 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.612668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b1673-4da0-4b3d-aadb-ffacccff92c2-tls-certs\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.612850 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.612717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-model-cache\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.612850 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.612751    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wd4\" (UniqueName: \"kubernetes.io/projected/269b1673-4da0-4b3d-aadb-ffacccff92c2-kube-api-access-d8wd4\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.612850 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.612772    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-tmp-dir\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.612850 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.612826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-home\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.613089 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.612867    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-dshm\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.613257 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.613224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-model-cache\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.613369 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.613262    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-home\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.613369 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.613283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-tmp-dir\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.615045 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.615012    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-dshm\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.615218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.615199    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b1673-4da0-4b3d-aadb-ffacccff92c2-tls-certs\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.620795 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.620774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wd4\" (UniqueName: \"kubernetes.io/projected/269b1673-4da0-4b3d-aadb-ffacccff92c2-kube-api-access-d8wd4\") pod \"autores-model-31f424-kserve-845ddbf896-rp6vd\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") " pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:16.734764 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:16.734711    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:17.003737 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:17.003712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"]
Sep 08 00:45:17.005340 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:45:17.005311    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod269b1673_4da0_4b3d_aadb_ffacccff92c2.slice/crio-f7d0ff861bb4347d5bc5da713614f05c54e40a478c546456afee3fc77caff4e1 WatchSource:0}: Error finding container f7d0ff861bb4347d5bc5da713614f05c54e40a478c546456afee3fc77caff4e1: Status 404 returned error can't find the container with id f7d0ff861bb4347d5bc5da713614f05c54e40a478c546456afee3fc77caff4e1
Sep 08 00:45:17.147675 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:17.147643    2555 scope.go:117] "RemoveContainer" containerID="406b1c255c7e34892afb175eb05aca0387aab210801f4bc45dc2fe666e060888"
Sep 08 00:45:17.167633 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:17.167580    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd" event={"ID":"269b1673-4da0-4b3d-aadb-ffacccff92c2","Type":"ContainerStarted","Data":"a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df"}
Sep 08 00:45:17.167763 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:17.167645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd" event={"ID":"269b1673-4da0-4b3d-aadb-ffacccff92c2","Type":"ContainerStarted","Data":"f7d0ff861bb4347d5bc5da713614f05c54e40a478c546456afee3fc77caff4e1"}
Sep 08 00:45:17.186542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:17.186494    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd" podStartSLOduration=1.186475899 podStartE2EDuration="1.186475899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:17.186320057 +0000 UTC m=+751.529623088" watchObservedRunningTime="2026-09-08 00:45:17.186475899 +0000 UTC m=+751.529778909"
Sep 08 00:45:17.735458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:17.735380    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:18.174648 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:18.174619    2555 generic.go:356] "Generic (PLEG): container finished" podID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerID="caae96570c85ae87b7f4a9b31a0be362f09eff1fe6fa90fcb12872d4dcc78fbc" exitCode=0
Sep 08 00:45:18.174648 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:18.174629    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm" event={"ID":"9cb765c3-9b95-4901-92d2-dcb6bdd8127c","Type":"ContainerDied","Data":"caae96570c85ae87b7f4a9b31a0be362f09eff1fe6fa90fcb12872d4dcc78fbc"}
Sep 08 00:45:18.174833 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:18.174689    2555 scope.go:117] "RemoveContainer" containerID="406b1c255c7e34892afb175eb05aca0387aab210801f4bc45dc2fe666e060888"
Sep 08 00:45:19.206970 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:19.206934    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"]
Sep 08 00:45:19.321075 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:19.321054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"
Sep 08 00:45:20.188831 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:20.188809    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"
Sep 08 00:45:20.188991 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:20.188829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm" event={"ID":"9cb765c3-9b95-4901-92d2-dcb6bdd8127c","Type":"ContainerDied","Data":"3697c54fc4810d84c6247b795e3913e4f2a992461eb40b3448ba0e65f73f3d42"}
Sep 08 00:45:20.188991 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:20.188881    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3697c54fc4810d84c6247b795e3913e4f2a992461eb40b3448ba0e65f73f3d42"
Sep 08 00:45:20.219670 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:20.219629    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"]
Sep 08 00:45:20.225192 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:20.225169    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f80520f6-29813805-t8xxm"]
Sep 08 00:45:22.153240 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:22.153206    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" path="/var/lib/kubelet/pods/9cb765c3-9b95-4901-92d2-dcb6bdd8127c/volumes"
Sep 08 00:45:23.646878 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:23.646833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:45:26.734839 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:26.734809    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:26.747404 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:26.747380    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:27.242685 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:27.242657    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:45:28.992619 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:28.992554    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"]
Sep 08 00:45:28.993138 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:28.992895    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx" podUID="34591cf7-32d0-4ee6-a9b8-64de2fb85f84" containerName="maas-api" containerID="cri-o://881e883050b033f3bc96ec52017d480422824d23fe2eeed5a86cfa5863c7826b" gracePeriod=30
Sep 08 00:45:29.244469 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.244393    2555 generic.go:356] "Generic (PLEG): container finished" podID="34591cf7-32d0-4ee6-a9b8-64de2fb85f84" containerID="881e883050b033f3bc96ec52017d480422824d23fe2eeed5a86cfa5863c7826b" exitCode=0
Sep 08 00:45:29.244645 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.244465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx" event={"ID":"34591cf7-32d0-4ee6-a9b8-64de2fb85f84","Type":"ContainerDied","Data":"881e883050b033f3bc96ec52017d480422824d23fe2eeed5a86cfa5863c7826b"}
Sep 08 00:45:29.244645 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.244544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx" event={"ID":"34591cf7-32d0-4ee6-a9b8-64de2fb85f84","Type":"ContainerDied","Data":"7be60f90ccd6600e9b9d62c144a152fe86d566528e2316f24ca0d06201faa7c8"}
Sep 08 00:45:29.244645 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.244571    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7be60f90ccd6600e9b9d62c144a152fe86d566528e2316f24ca0d06201faa7c8"
Sep 08 00:45:29.248757 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.248739    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:45:29.422892 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.422852    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-maas-api-tls\") pod \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") "
Sep 08 00:45:29.422892 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.422892    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs6ng\" (UniqueName: \"kubernetes.io/projected/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-kube-api-access-vs6ng\") pod \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") "
Sep 08 00:45:29.423130 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.422945    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-metrics-certs\") pod \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") "
Sep 08 00:45:29.423130 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.422979    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-openshift-service-ca\") pod \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\" (UID: \"34591cf7-32d0-4ee6-a9b8-64de2fb85f84\") "
Sep 08 00:45:29.423409 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.423378    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34591cf7-32d0-4ee6-a9b8-64de2fb85f84" (UID: "34591cf7-32d0-4ee6-a9b8-64de2fb85f84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:45:29.425395 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.425368    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34591cf7-32d0-4ee6-a9b8-64de2fb85f84" (UID: "34591cf7-32d0-4ee6-a9b8-64de2fb85f84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:45:29.425585 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.425565    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-kube-api-access-vs6ng" (OuterVolumeSpecName: "kube-api-access-vs6ng") pod "34591cf7-32d0-4ee6-a9b8-64de2fb85f84" (UID: "34591cf7-32d0-4ee6-a9b8-64de2fb85f84"). InnerVolumeSpecName "kube-api-access-vs6ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:45:29.425671 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.425588    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34591cf7-32d0-4ee6-a9b8-64de2fb85f84" (UID: "34591cf7-32d0-4ee6-a9b8-64de2fb85f84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:45:29.524426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.524370    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:45:29.524426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.524393    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6ng\" (UniqueName: \"kubernetes.io/projected/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-kube-api-access-vs6ng\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:45:29.524426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.524404    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:45:29.524426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:29.524413    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34591cf7-32d0-4ee6-a9b8-64de2fb85f84-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:45:30.250439 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:30.250409    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"
Sep 08 00:45:30.268062 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:30.268033    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"]
Sep 08 00:45:30.272555 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:30.272530    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f80520f6-7689c75866-dzpwx"]
Sep 08 00:45:32.153791 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:32.153746    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34591cf7-32d0-4ee6-a9b8-64de2fb85f84" path="/var/lib/kubelet/pods/34591cf7-32d0-4ee6-a9b8-64de2fb85f84/volumes"
Sep 08 00:45:34.826063 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826026    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"]
Sep 08 00:45:34.826766 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826730    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:45:34.826766 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826762    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:45:34.826941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826792    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34591cf7-32d0-4ee6-a9b8-64de2fb85f84" containerName="maas-api"
Sep 08 00:45:34.826941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826801    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="34591cf7-32d0-4ee6-a9b8-64de2fb85f84" containerName="maas-api"
Sep 08 00:45:34.826941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826820    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:45:34.826941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826830    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:45:34.826941 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826935    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:45:34.827149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826951    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:45:34.827149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.826962    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="34591cf7-32d0-4ee6-a9b8-64de2fb85f84" containerName="maas-api"
Sep 08 00:45:34.831908 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.831885    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.834506 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.834478    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9b001d57-serving-cert\""
Sep 08 00:45:34.839542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.839520    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"]
Sep 08 00:45:34.870673 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.870646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-metrics-certs\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.870791 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.870686    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ef6243-5dbf-4262-abac-b4dc4cefcf83-openshift-service-ca\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.870845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.870785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2btl\" (UniqueName: \"kubernetes.io/projected/79ef6243-5dbf-4262-abac-b4dc4cefcf83-kube-api-access-d2btl\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.870845 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.870813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-maas-api-tls\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.971547 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.971514    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2btl\" (UniqueName: \"kubernetes.io/projected/79ef6243-5dbf-4262-abac-b4dc4cefcf83-kube-api-access-d2btl\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.971547 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.971543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-maas-api-tls\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.971734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.971575    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-metrics-certs\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.971734 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.971614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ef6243-5dbf-4262-abac-b4dc4cefcf83-openshift-service-ca\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.972150 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.972126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ef6243-5dbf-4262-abac-b4dc4cefcf83-openshift-service-ca\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.973850 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.973817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-metrics-certs\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.973984 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.973964    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-maas-api-tls\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:34.979624 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:34.979586    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2btl\" (UniqueName: \"kubernetes.io/projected/79ef6243-5dbf-4262-abac-b4dc4cefcf83-kube-api-access-d2btl\") pod \"maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:35.146185 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:35.146113    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:35.418252 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:35.418224    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"]
Sep 08 00:45:35.420107 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:45:35.420077    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ef6243_5dbf_4262_abac_b4dc4cefcf83.slice/crio-cd5ee5ec7dc38c8a8ccd3d917076ea90bbe3636d6cc5c30076606fbdf1b82a7c WatchSource:0}: Error finding container cd5ee5ec7dc38c8a8ccd3d917076ea90bbe3636d6cc5c30076606fbdf1b82a7c: Status 404 returned error can't find the container with id cd5ee5ec7dc38c8a8ccd3d917076ea90bbe3636d6cc5c30076606fbdf1b82a7c
Sep 08 00:45:36.284714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:36.284627    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr" event={"ID":"79ef6243-5dbf-4262-abac-b4dc4cefcf83","Type":"ContainerStarted","Data":"113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73"}
Sep 08 00:45:36.284714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:36.284696    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:36.284714 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:36.284716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr" event={"ID":"79ef6243-5dbf-4262-abac-b4dc4cefcf83","Type":"ContainerStarted","Data":"cd5ee5ec7dc38c8a8ccd3d917076ea90bbe3636d6cc5c30076606fbdf1b82a7c"}
Sep 08 00:45:36.302855 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:36.302805    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr" podStartSLOduration=1.796979518 podStartE2EDuration="2.302791728s" totalImagesPullingTime="505.81221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:36.300861311 +0000 UTC m=+770.644164331" watchObservedRunningTime="2026-09-08 00:45:36.302791728 +0000 UTC m=+770.646094747"
Sep 08 00:45:42.295949 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:42.295918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:45:49.136992 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:45:49.136951    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:46:05.501171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.501135    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"]
Sep 08 00:46:05.501571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.501336    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr" podUID="79ef6243-5dbf-4262-abac-b4dc4cefcf83" containerName="maas-api" containerID="cri-o://113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73" gracePeriod=30
Sep 08 00:46:05.743230 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.743208    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:46:05.847634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.847605    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2btl\" (UniqueName: \"kubernetes.io/projected/79ef6243-5dbf-4262-abac-b4dc4cefcf83-kube-api-access-d2btl\") pod \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") "
Sep 08 00:46:05.847781 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.847661    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-maas-api-tls\") pod \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") "
Sep 08 00:46:05.847781 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.847711    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ef6243-5dbf-4262-abac-b4dc4cefcf83-openshift-service-ca\") pod \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") "
Sep 08 00:46:05.847781 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.847737    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-metrics-certs\") pod \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\" (UID: \"79ef6243-5dbf-4262-abac-b4dc4cefcf83\") "
Sep 08 00:46:05.848145 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.848121    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ef6243-5dbf-4262-abac-b4dc4cefcf83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79ef6243-5dbf-4262-abac-b4dc4cefcf83" (UID: "79ef6243-5dbf-4262-abac-b4dc4cefcf83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:46:05.850458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.850430    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79ef6243-5dbf-4262-abac-b4dc4cefcf83" (UID: "79ef6243-5dbf-4262-abac-b4dc4cefcf83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:46:05.851898 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.851853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79ef6243-5dbf-4262-abac-b4dc4cefcf83" (UID: "79ef6243-5dbf-4262-abac-b4dc4cefcf83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:46:05.852028 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.851998    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ef6243-5dbf-4262-abac-b4dc4cefcf83-kube-api-access-d2btl" (OuterVolumeSpecName: "kube-api-access-d2btl") pod "79ef6243-5dbf-4262-abac-b4dc4cefcf83" (UID: "79ef6243-5dbf-4262-abac-b4dc4cefcf83"). InnerVolumeSpecName "kube-api-access-d2btl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:46:05.948806 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.948775    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:05.948806 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.948803    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2btl\" (UniqueName: \"kubernetes.io/projected/79ef6243-5dbf-4262-abac-b4dc4cefcf83-kube-api-access-d2btl\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:05.948970 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.948813    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79ef6243-5dbf-4262-abac-b4dc4cefcf83-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:05.948970 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:05.948821    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ef6243-5dbf-4262-abac-b4dc4cefcf83-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:06.457946 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.457853    2555 generic.go:356] "Generic (PLEG): container finished" podID="79ef6243-5dbf-4262-abac-b4dc4cefcf83" containerID="113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73" exitCode=0
Sep 08 00:46:06.457946 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.457901    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"
Sep 08 00:46:06.458118 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.457935    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr" event={"ID":"79ef6243-5dbf-4262-abac-b4dc4cefcf83","Type":"ContainerDied","Data":"113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73"}
Sep 08 00:46:06.458118 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.457991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr" event={"ID":"79ef6243-5dbf-4262-abac-b4dc4cefcf83","Type":"ContainerDied","Data":"cd5ee5ec7dc38c8a8ccd3d917076ea90bbe3636d6cc5c30076606fbdf1b82a7c"}
Sep 08 00:46:06.458118 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.458019    2555 scope.go:117] "RemoveContainer" containerID="113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73"
Sep 08 00:46:06.467391 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.467372    2555 scope.go:117] "RemoveContainer" containerID="113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73"
Sep 08 00:46:06.467674 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:46:06.467657    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73\": container with ID starting with 113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73 not found: ID does not exist" containerID="113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73"
Sep 08 00:46:06.467730 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.467681    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73"} err="failed to get container status \"113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73\": rpc error: code = NotFound desc = could not find container \"113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73\": container with ID starting with 113999309f018fb299fce5bc1842bdf8d760e3391d27f197b7082568b6479a73 not found: ID does not exist"
Sep 08 00:46:06.486798 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.486760    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"]
Sep 08 00:46:06.490761 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:06.490736    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9b001d57-68cc9f44cb-lvpfr"]
Sep 08 00:46:07.716389 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:07.716351    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"]
Sep 08 00:46:08.154242 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:08.154211    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ef6243-5dbf-4262-abac-b4dc4cefcf83" path="/var/lib/kubelet/pods/79ef6243-5dbf-4262-abac-b4dc4cefcf83/volumes"
Sep 08 00:46:21.428002 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:21.427965    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:46:22.736281 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:22.736243    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd" podUID="269b1673-4da0-4b3d-aadb-ffacccff92c2" containerName="main" containerID="cri-o://a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df" gracePeriod=45
Sep 08 00:46:22.992238 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:22.992185    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:46:23.093549 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.093515    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-dshm\") pod \"269b1673-4da0-4b3d-aadb-ffacccff92c2\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") "
Sep 08 00:46:23.093742 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.093558    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8wd4\" (UniqueName: \"kubernetes.io/projected/269b1673-4da0-4b3d-aadb-ffacccff92c2-kube-api-access-d8wd4\") pod \"269b1673-4da0-4b3d-aadb-ffacccff92c2\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") "
Sep 08 00:46:23.093742 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.093635    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b1673-4da0-4b3d-aadb-ffacccff92c2-tls-certs\") pod \"269b1673-4da0-4b3d-aadb-ffacccff92c2\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") "
Sep 08 00:46:23.093742 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.093735    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-model-cache\") pod \"269b1673-4da0-4b3d-aadb-ffacccff92c2\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") "
Sep 08 00:46:23.093928 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.093773    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-tmp-dir\") pod \"269b1673-4da0-4b3d-aadb-ffacccff92c2\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") "
Sep 08 00:46:23.093928 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.093866    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-home\") pod \"269b1673-4da0-4b3d-aadb-ffacccff92c2\" (UID: \"269b1673-4da0-4b3d-aadb-ffacccff92c2\") "
Sep 08 00:46:23.094028 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.093981    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "269b1673-4da0-4b3d-aadb-ffacccff92c2" (UID: "269b1673-4da0-4b3d-aadb-ffacccff92c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:46:23.094079 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.094059    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "269b1673-4da0-4b3d-aadb-ffacccff92c2" (UID: "269b1673-4da0-4b3d-aadb-ffacccff92c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:46:23.094191 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.094164    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-home" (OuterVolumeSpecName: "home") pod "269b1673-4da0-4b3d-aadb-ffacccff92c2" (UID: "269b1673-4da0-4b3d-aadb-ffacccff92c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:46:23.094247 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.094235    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:23.094283 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.094252    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:23.094283 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.094260    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:23.095715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.095694    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-dshm" (OuterVolumeSpecName: "dshm") pod "269b1673-4da0-4b3d-aadb-ffacccff92c2" (UID: "269b1673-4da0-4b3d-aadb-ffacccff92c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:46:23.095805 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.095748    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269b1673-4da0-4b3d-aadb-ffacccff92c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "269b1673-4da0-4b3d-aadb-ffacccff92c2" (UID: "269b1673-4da0-4b3d-aadb-ffacccff92c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:46:23.095876 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.095862    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269b1673-4da0-4b3d-aadb-ffacccff92c2-kube-api-access-d8wd4" (OuterVolumeSpecName: "kube-api-access-d8wd4") pod "269b1673-4da0-4b3d-aadb-ffacccff92c2" (UID: "269b1673-4da0-4b3d-aadb-ffacccff92c2"). InnerVolumeSpecName "kube-api-access-d8wd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:46:23.195525 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.195502    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/269b1673-4da0-4b3d-aadb-ffacccff92c2-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:23.195525 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.195522    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8wd4\" (UniqueName: \"kubernetes.io/projected/269b1673-4da0-4b3d-aadb-ffacccff92c2-kube-api-access-d8wd4\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:23.195674 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.195533    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/269b1673-4da0-4b3d-aadb-ffacccff92c2-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:46:23.554370 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.554335    2555 generic.go:356] "Generic (PLEG): container finished" podID="269b1673-4da0-4b3d-aadb-ffacccff92c2" containerID="a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df" exitCode=0
Sep 08 00:46:23.554548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.554384    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"
Sep 08 00:46:23.554548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.554415    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd" event={"ID":"269b1673-4da0-4b3d-aadb-ffacccff92c2","Type":"ContainerDied","Data":"a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df"}
Sep 08 00:46:23.554548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.554453    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd" event={"ID":"269b1673-4da0-4b3d-aadb-ffacccff92c2","Type":"ContainerDied","Data":"f7d0ff861bb4347d5bc5da713614f05c54e40a478c546456afee3fc77caff4e1"}
Sep 08 00:46:23.554548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.554476    2555 scope.go:117] "RemoveContainer" containerID="a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df"
Sep 08 00:46:23.564091 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.564071    2555 scope.go:117] "RemoveContainer" containerID="a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df"
Sep 08 00:46:23.564371 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:46:23.564347    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df\": container with ID starting with a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df not found: ID does not exist" containerID="a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df"
Sep 08 00:46:23.564435 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.564383    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df"} err="failed to get container status \"a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df\": rpc error: code = NotFound desc = could not find container \"a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df\": container with ID starting with a85d82f2b7068c1cbbdefb77d759228779d3b82283e8be18f70cc502401811df not found: ID does not exist"
Sep 08 00:46:23.578764 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.578735    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"]
Sep 08 00:46:23.586198 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:23.586173    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-31f424/autores-model-31f424-kserve-845ddbf896-rp6vd"]
Sep 08 00:46:24.153004 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:24.152970    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="269b1673-4da0-4b3d-aadb-ffacccff92c2" path="/var/lib/kubelet/pods/269b1673-4da0-4b3d-aadb-ffacccff92c2/volumes"
Sep 08 00:46:36.531175 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:36.531138    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:46:37.717313 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717277    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"]
Sep 08 00:46:37.717726 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717712    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ef6243-5dbf-4262-abac-b4dc4cefcf83" containerName="maas-api"
Sep 08 00:46:37.717726 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717725    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ef6243-5dbf-4262-abac-b4dc4cefcf83" containerName="maas-api"
Sep 08 00:46:37.717809 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717737    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:46:37.717809 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717743    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:46:37.717809 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717762    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269b1673-4da0-4b3d-aadb-ffacccff92c2" containerName="main"
Sep 08 00:46:37.717809 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717768    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="269b1673-4da0-4b3d-aadb-ffacccff92c2" containerName="main"
Sep 08 00:46:37.717953 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717831    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cb765c3-9b95-4901-92d2-dcb6bdd8127c" containerName="cleanup"
Sep 08 00:46:37.717953 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717840    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ef6243-5dbf-4262-abac-b4dc4cefcf83" containerName="maas-api"
Sep 08 00:46:37.717953 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.717850    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="269b1673-4da0-4b3d-aadb-ffacccff92c2" containerName="main"
Sep 08 00:46:37.722060 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.722037    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.724375 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.724355    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-4a8d93c0-kserve-self-signed-certs\""
Sep 08 00:46:37.730672 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.730646    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"]
Sep 08 00:46:37.829276 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.829244    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-dshm\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.829435 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.829342    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-home\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.829435 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.829389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdvr\" (UniqueName: \"kubernetes.io/projected/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-kube-api-access-kkdvr\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.829435 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.829426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tmp-dir\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.829564 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.829442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tls-certs\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.829564 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.829511    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-model-cache\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930274 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdvr\" (UniqueName: \"kubernetes.io/projected/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-kube-api-access-kkdvr\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930457 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tmp-dir\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930457 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tls-certs\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930457 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-model-cache\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930457 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-dshm\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930457 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930397    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-home\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930773 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930748    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tmp-dir\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930836 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-model-cache\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.930894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.930845    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-home\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.932622 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.932581    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-dshm\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.932849 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.932830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tls-certs\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:37.937733 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:37.937711    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdvr\" (UniqueName: \"kubernetes.io/projected/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-kube-api-access-kkdvr\") pod \"e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") " pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:38.031410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:38.031340    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:38.302152 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:38.302096    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"]
Sep 08 00:46:38.304953 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:46:38.304925    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc289fc4_0b2a_4d10_a707_b19acb6e8c33.slice/crio-c1e0d72e4d9fd53c75585a59de361f90663e2b6220a06a836094a0a28569a14b WatchSource:0}: Error finding container c1e0d72e4d9fd53c75585a59de361f90663e2b6220a06a836094a0a28569a14b: Status 404 returned error can't find the container with id c1e0d72e4d9fd53c75585a59de361f90663e2b6220a06a836094a0a28569a14b
Sep 08 00:46:38.648026 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:38.647933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6" event={"ID":"fc289fc4-0b2a-4d10-a707-b19acb6e8c33","Type":"ContainerStarted","Data":"5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f"}
Sep 08 00:46:38.648026 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:38.647980    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6" event={"ID":"fc289fc4-0b2a-4d10-a707-b19acb6e8c33","Type":"ContainerStarted","Data":"c1e0d72e4d9fd53c75585a59de361f90663e2b6220a06a836094a0a28569a14b"}
Sep 08 00:46:38.665358 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:38.665310    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6" podStartSLOduration=1.665296954 podStartE2EDuration="1.665296954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:46:38.663787717 +0000 UTC m=+833.007090737" watchObservedRunningTime="2026-09-08 00:46:38.665296954 +0000 UTC m=+833.008599972"
Sep 08 00:46:39.032462 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:39.032383    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:48.032322 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:48.032278    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:48.044929 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:48.044899    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:48.711607 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:48.711555    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:46:53.831280 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:46:53.831236    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:47:07.216724 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.216690    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"]
Sep 08 00:47:07.222089 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.222066    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.225917 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.225892    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1cf84f\"/\"default-dockercfg-lwg5s\""
Sep 08 00:47:07.225917 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.225895    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1cf84f\"/\"test-model-1cf84f-kserve-self-signed-certs\""
Sep 08 00:47:07.226123 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.225945    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1cf84f\"/\"kube-root-ca.crt\""
Sep 08 00:47:07.226123 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.226019    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1cf84f\"/\"openshift-service-ca.crt\""
Sep 08 00:47:07.231008 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.230987    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"]
Sep 08 00:47:07.301052 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.301014    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-tmp-dir\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.301236 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.301060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-home\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.301236 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.301083    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-dshm\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.301236 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.301139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d06118b-e985-4c14-87c7-a9200342b34d-tls-certs\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.301236 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.301218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92tg\" (UniqueName: \"kubernetes.io/projected/3d06118b-e985-4c14-87c7-a9200342b34d-kube-api-access-k92tg\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.301439 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.301242    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-model-cache\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402202 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-home\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402379 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402209    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-dshm\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402379 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d06118b-e985-4c14-87c7-a9200342b34d-tls-certs\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402379 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402291    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92tg\" (UniqueName: \"kubernetes.io/projected/3d06118b-e985-4c14-87c7-a9200342b34d-kube-api-access-k92tg\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402379 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-model-cache\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402630 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402435    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-tmp-dir\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402698 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-home\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402761 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-model-cache\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.402799 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.402780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-tmp-dir\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.404547 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.404527    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-dshm\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.404840 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.404822    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d06118b-e985-4c14-87c7-a9200342b34d-tls-certs\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.410014 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.409988    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92tg\" (UniqueName: \"kubernetes.io/projected/3d06118b-e985-4c14-87c7-a9200342b34d-kube-api-access-k92tg\") pod \"test-model-1cf84f-kserve-684564f7db-f8sjl\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") " pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.533521 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.533422    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:07.802763 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:07.802737    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"]
Sep 08 00:47:07.805053 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:47:07.805027    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d06118b_e985_4c14_87c7_a9200342b34d.slice/crio-3c12b901d21a73c3f8475b5a390d2d9a54032ec9d200af3744ceb6c19aa10ff1 WatchSource:0}: Error finding container 3c12b901d21a73c3f8475b5a390d2d9a54032ec9d200af3744ceb6c19aa10ff1: Status 404 returned error can't find the container with id 3c12b901d21a73c3f8475b5a390d2d9a54032ec9d200af3744ceb6c19aa10ff1
Sep 08 00:47:08.817996 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:08.817954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl" event={"ID":"3d06118b-e985-4c14-87c7-a9200342b34d","Type":"ContainerStarted","Data":"045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1"}
Sep 08 00:47:08.818372 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:08.818004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl" event={"ID":"3d06118b-e985-4c14-87c7-a9200342b34d","Type":"ContainerStarted","Data":"3c12b901d21a73c3f8475b5a390d2d9a54032ec9d200af3744ceb6c19aa10ff1"}
Sep 08 00:47:08.850870 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:08.850814    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl" podStartSLOduration=1.850800614 podStartE2EDuration="1.850800614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:47:08.850456779 +0000 UTC m=+863.193759793" watchObservedRunningTime="2026-09-08 00:47:08.850800614 +0000 UTC m=+863.194103624"
Sep 08 00:47:09.533725 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:09.533679    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:12.921024 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:12.920989    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"]
Sep 08 00:47:17.533847 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:17.533803    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:17.546969 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:17.546942    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:17.887445 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:17.887415    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:47:19.864888 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:19.864845    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:47:27.941205 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:27.941165    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6" podUID="fc289fc4-0b2a-4d10-a707-b19acb6e8c33" containerName="main" containerID="cri-o://5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f" gracePeriod=45
Sep 08 00:47:28.198159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.198098    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:47:28.288308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288281    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-model-cache\") pod \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") "
Sep 08 00:47:28.288461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288340    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tls-certs\") pod \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") "
Sep 08 00:47:28.288461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288378    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-dshm\") pod \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") "
Sep 08 00:47:28.288461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288414    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkdvr\" (UniqueName: \"kubernetes.io/projected/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-kube-api-access-kkdvr\") pod \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") "
Sep 08 00:47:28.288461 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288429    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tmp-dir\") pod \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") "
Sep 08 00:47:28.288679 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288467    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-home\") pod \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\" (UID: \"fc289fc4-0b2a-4d10-a707-b19acb6e8c33\") "
Sep 08 00:47:28.288679 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288573    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc289fc4-0b2a-4d10-a707-b19acb6e8c33" (UID: "fc289fc4-0b2a-4d10-a707-b19acb6e8c33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:47:28.288786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288711    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc289fc4-0b2a-4d10-a707-b19acb6e8c33" (UID: "fc289fc4-0b2a-4d10-a707-b19acb6e8c33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:47:28.288842 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288802    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:47:28.288842 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.288820    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:47:28.289084 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.289040    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-home" (OuterVolumeSpecName: "home") pod "fc289fc4-0b2a-4d10-a707-b19acb6e8c33" (UID: "fc289fc4-0b2a-4d10-a707-b19acb6e8c33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:47:28.290540 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.290522    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc289fc4-0b2a-4d10-a707-b19acb6e8c33" (UID: "fc289fc4-0b2a-4d10-a707-b19acb6e8c33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:47:28.290894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.290877    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-dshm" (OuterVolumeSpecName: "dshm") pod "fc289fc4-0b2a-4d10-a707-b19acb6e8c33" (UID: "fc289fc4-0b2a-4d10-a707-b19acb6e8c33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:47:28.290951 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.290930    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-kube-api-access-kkdvr" (OuterVolumeSpecName: "kube-api-access-kkdvr") pod "fc289fc4-0b2a-4d10-a707-b19acb6e8c33" (UID: "fc289fc4-0b2a-4d10-a707-b19acb6e8c33"). InnerVolumeSpecName "kube-api-access-kkdvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:47:28.389410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.389379    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:47:28.389410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.389403    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:47:28.389410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.389413    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkdvr\" (UniqueName: \"kubernetes.io/projected/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-kube-api-access-kkdvr\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:47:28.389634 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.389422    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc289fc4-0b2a-4d10-a707-b19acb6e8c33-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:47:28.947411 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.947374    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc289fc4-0b2a-4d10-a707-b19acb6e8c33" containerID="5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f" exitCode=0
Sep 08 00:47:28.947813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.947430    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"
Sep 08 00:47:28.947813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.947462    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6" event={"ID":"fc289fc4-0b2a-4d10-a707-b19acb6e8c33","Type":"ContainerDied","Data":"5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f"}
Sep 08 00:47:28.947813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.947507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6" event={"ID":"fc289fc4-0b2a-4d10-a707-b19acb6e8c33","Type":"ContainerDied","Data":"c1e0d72e4d9fd53c75585a59de361f90663e2b6220a06a836094a0a28569a14b"}
Sep 08 00:47:28.947813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.947526    2555 scope.go:117] "RemoveContainer" containerID="5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f"
Sep 08 00:47:28.957710 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.957693    2555 scope.go:117] "RemoveContainer" containerID="5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f"
Sep 08 00:47:28.957979 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:47:28.957959    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f\": container with ID starting with 5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f not found: ID does not exist" containerID="5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f"
Sep 08 00:47:28.958051 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.957988    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f"} err="failed to get container status \"5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f\": rpc error: code = NotFound desc = could not find container \"5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f\": container with ID starting with 5ce45bc64d6c91c07e9fd7a98eeb45e7ef21a591b2c254480295554d2d71b42f not found: ID does not exist"
Sep 08 00:47:28.969902 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.969879    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"]
Sep 08 00:47:28.977739 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:28.977720    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-4a8d93c0-kserve-c7b885d6d-fz9z6"]
Sep 08 00:47:30.153433 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:30.153399    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc289fc4-0b2a-4d10-a707-b19acb6e8c33" path="/var/lib/kubelet/pods/fc289fc4-0b2a-4d10-a707-b19acb6e8c33/volumes"
Sep 08 00:47:45.024240 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.024151    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"]
Sep 08 00:47:45.024984 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.024963    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc289fc4-0b2a-4d10-a707-b19acb6e8c33" containerName="main"
Sep 08 00:47:45.025069 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.024987    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc289fc4-0b2a-4d10-a707-b19acb6e8c33" containerName="main"
Sep 08 00:47:45.025109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.025081    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc289fc4-0b2a-4d10-a707-b19acb6e8c33" containerName="main"
Sep 08 00:47:45.028656 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.028634    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.031111 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.031087    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f627f164-serving-cert\""
Sep 08 00:47:45.039946 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.039883    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"]
Sep 08 00:47:45.142852 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.142814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.143032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.142873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-metrics-certs\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.143032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.142919    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-openshift-service-ca\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.143032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.142958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdsp\" (UniqueName: \"kubernetes.io/projected/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-kube-api-access-7hdsp\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.244082 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.244035    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-metrics-certs\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.244273 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.244117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-openshift-service-ca\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.244273 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.244175    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdsp\" (UniqueName: \"kubernetes.io/projected/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-kube-api-access-7hdsp\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.244394 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.244292    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.244513 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:47:45.244484    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-serving-cert: secret "maas-api-e2e-mt-f627f164-serving-cert" not found
Sep 08 00:47:45.244669 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:47:45.244587    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls podName:fc7eadd3-eaee-492b-8b64-4fe6b3a36422 nodeName:}" failed. No retries permitted until 2026-09-08 00:47:45.744563358 +0000 UTC m=+900.087866358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls") pod "maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw" (UID: "fc7eadd3-eaee-492b-8b64-4fe6b3a36422") : secret "maas-api-e2e-mt-f627f164-serving-cert" not found
Sep 08 00:47:45.244896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.244873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-openshift-service-ca\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.246948 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.246927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-metrics-certs\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.255384 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.255362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdsp\" (UniqueName: \"kubernetes.io/projected/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-kube-api-access-7hdsp\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.747476 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.747429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.750130 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.750094    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls\") pod \"maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:45.963513 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:45.963476    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:46.152777 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:46.152739    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:47:46.153227 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:46.153044    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:47:46.238295 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:46.238257    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"]
Sep 08 00:47:46.240040 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:47:46.240006    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7eadd3_eaee_492b_8b64_4fe6b3a36422.slice/crio-3823563a29da6c2af1d35fcfa2ce0beb5a1893ff500bbd54a906fcba7ec787bf WatchSource:0}: Error finding container 3823563a29da6c2af1d35fcfa2ce0beb5a1893ff500bbd54a906fcba7ec787bf: Status 404 returned error can't find the container with id 3823563a29da6c2af1d35fcfa2ce0beb5a1893ff500bbd54a906fcba7ec787bf
Sep 08 00:47:47.051247 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:47.051207    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw" event={"ID":"fc7eadd3-eaee-492b-8b64-4fe6b3a36422","Type":"ContainerStarted","Data":"fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71"}
Sep 08 00:47:47.051463 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:47.051269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw" event={"ID":"fc7eadd3-eaee-492b-8b64-4fe6b3a36422","Type":"ContainerStarted","Data":"3823563a29da6c2af1d35fcfa2ce0beb5a1893ff500bbd54a906fcba7ec787bf"}
Sep 08 00:47:47.051463 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:47.051323    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:47:47.073074 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:47.073016    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw" podStartSLOduration=1.7063810579999998 podStartE2EDuration="2.073000162s" totalImagesPullingTime="366.619104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:47:47.071657011 +0000 UTC m=+901.414960049" watchObservedRunningTime="2026-09-08 00:47:47.073000162 +0000 UTC m=+901.416303181"
Sep 08 00:47:51.130987 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:51.130947    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:47:53.062171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:47:53.062131    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:48:06.750690 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.750654    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"]
Sep 08 00:48:06.755003 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.754947    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.758813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.758791    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e3896b\"/\"openshift-service-ca.crt\""
Sep 08 00:48:06.758939 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.758822    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e3896b\"/\"test-model-e3896b-kserve-self-signed-certs\""
Sep 08 00:48:06.758939 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.758882    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e3896b\"/\"default-dockercfg-sbzkp\""
Sep 08 00:48:06.758939 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.758921    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e3896b\"/\"kube-root-ca.crt\""
Sep 08 00:48:06.762932 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.762910    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"]
Sep 08 00:48:06.826529 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.826492    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-home\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.826529 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.826531    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zk7t\" (UniqueName: \"kubernetes.io/projected/e47d5a24-685f-49c4-8d49-5da84e4cfe73-kube-api-access-4zk7t\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.826744 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.826587    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tmp-dir\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.826744 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.826676    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tls-certs\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.826816 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.826742    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-dshm\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.826816 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.826762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-model-cache\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.927696 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.927665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-home\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.927696 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.927698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zk7t\" (UniqueName: \"kubernetes.io/projected/e47d5a24-685f-49c4-8d49-5da84e4cfe73-kube-api-access-4zk7t\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.927930 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.927734    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tmp-dir\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.927930 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.927770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tls-certs\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.927930 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.927823    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-dshm\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.927930 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.927848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-model-cache\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.928186 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.928148    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-home\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.928186 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.928184    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tmp-dir\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.928362 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.928203    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-model-cache\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.930113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.930087    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-dshm\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.930283 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.930262    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tls-certs\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:06.935062 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:06.935042    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zk7t\" (UniqueName: \"kubernetes.io/projected/e47d5a24-685f-49c4-8d49-5da84e4cfe73-kube-api-access-4zk7t\") pod \"test-model-e3896b-kserve-bd876dd88-68dtp\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") " pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:07.064935 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:07.064896    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:07.339625 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:07.339576    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"]
Sep 08 00:48:07.339998 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:48:07.339974    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode47d5a24_685f_49c4_8d49_5da84e4cfe73.slice/crio-f5e45bedf038b89ca47a9c3bf63f49782deccdf7ce568206428c58eca1d6acfb WatchSource:0}: Error finding container f5e45bedf038b89ca47a9c3bf63f49782deccdf7ce568206428c58eca1d6acfb: Status 404 returned error can't find the container with id f5e45bedf038b89ca47a9c3bf63f49782deccdf7ce568206428c58eca1d6acfb
Sep 08 00:48:08.185462 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:08.185420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp" event={"ID":"e47d5a24-685f-49c4-8d49-5da84e4cfe73","Type":"ContainerStarted","Data":"b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4"}
Sep 08 00:48:08.185869 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:08.185473    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp" event={"ID":"e47d5a24-685f-49c4-8d49-5da84e4cfe73","Type":"ContainerStarted","Data":"f5e45bedf038b89ca47a9c3bf63f49782deccdf7ce568206428c58eca1d6acfb"}
Sep 08 00:48:08.203358 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:08.203265    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp" podStartSLOduration=2.203236898 podStartE2EDuration="2.203236898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:48:08.201003583 +0000 UTC m=+922.544306598" watchObservedRunningTime="2026-09-08 00:48:08.203236898 +0000 UTC m=+922.546539908"
Sep 08 00:48:09.065938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:09.065903    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:16.739239 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.739188    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:48:16.832084 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.832050    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"]
Sep 08 00:48:16.835896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.835873    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:16.839838 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.839797    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e4902f\"/\"kube-root-ca.crt\""
Sep 08 00:48:16.839996 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.839836    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e4902f\"/\"openshift-service-ca.crt\""
Sep 08 00:48:16.839996 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.839874    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e4902f\"/\"default-dockercfg-58bxk\""
Sep 08 00:48:16.839996 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.839797    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e4902f\"/\"explicit-model-e4902f-kserve-self-signed-certs\""
Sep 08 00:48:16.848506 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.848463    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"]
Sep 08 00:48:16.921196 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.921150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6wn\" (UniqueName: \"kubernetes.io/projected/1753be71-bd43-47cf-9e2d-b32d151fac8c-kube-api-access-hs6wn\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:16.921388 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.921228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753be71-bd43-47cf-9e2d-b32d151fac8c-tls-certs\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:16.921388 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.921324    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-dshm\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:16.921388 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.921354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-tmp-dir\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:16.921388 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.921378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-home\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:16.921544 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:16.921428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-model-cache\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022059 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.021956    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6wn\" (UniqueName: \"kubernetes.io/projected/1753be71-bd43-47cf-9e2d-b32d151fac8c-kube-api-access-hs6wn\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022059 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022019    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753be71-bd43-47cf-9e2d-b32d151fac8c-tls-certs\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022302 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-dshm\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022302 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022120    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-tmp-dir\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022302 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-home\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022302 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022198    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-model-cache\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022655 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022622    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-tmp-dir\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022796 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022639    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-home\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.022796 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.022710    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-model-cache\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.024688 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.024666    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-dshm\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.024856 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.024828    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753be71-bd43-47cf-9e2d-b32d151fac8c-tls-certs\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.031134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.031109    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6wn\" (UniqueName: \"kubernetes.io/projected/1753be71-bd43-47cf-9e2d-b32d151fac8c-kube-api-access-hs6wn\") pod \"explicit-model-e4902f-kserve-68f5fc677d-7kq96\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") " pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.066024 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.065984    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:17.078886 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.078849    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:17.150000 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.149958    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:17.252913 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.252859    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:48:17.424929 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:17.424893    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"]
Sep 08 00:48:17.425705 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:48:17.425670    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1753be71_bd43_47cf_9e2d_b32d151fac8c.slice/crio-07c36f402efeaff0cf573766ebdedd77e970479c563bc62e4e5dd92a23894399 WatchSource:0}: Error finding container 07c36f402efeaff0cf573766ebdedd77e970479c563bc62e4e5dd92a23894399: Status 404 returned error can't find the container with id 07c36f402efeaff0cf573766ebdedd77e970479c563bc62e4e5dd92a23894399
Sep 08 00:48:18.248791 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:18.248749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96" event={"ID":"1753be71-bd43-47cf-9e2d-b32d151fac8c","Type":"ContainerStarted","Data":"1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d"}
Sep 08 00:48:18.249257 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:18.248807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96" event={"ID":"1753be71-bd43-47cf-9e2d-b32d151fac8c","Type":"ContainerStarted","Data":"07c36f402efeaff0cf573766ebdedd77e970479c563bc62e4e5dd92a23894399"}
Sep 08 00:48:18.266677 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:18.266629    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96" podStartSLOduration=2.266613166 podStartE2EDuration="2.266613166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:48:18.264855589 +0000 UTC m=+932.608158606" watchObservedRunningTime="2026-09-08 00:48:18.266613166 +0000 UTC m=+932.609916184"
Sep 08 00:48:19.150543 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:19.150513    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:26.482378 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.481573    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"]
Sep 08 00:48:26.482378 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.481821    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw" podUID="fc7eadd3-eaee-492b-8b64-4fe6b3a36422" containerName="maas-api" containerID="cri-o://fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71" gracePeriod=30
Sep 08 00:48:26.738947 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.738891    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:48:26.815855 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.815824    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls\") pod \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") "
Sep 08 00:48:26.815993 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.815877    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-openshift-service-ca\") pod \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") "
Sep 08 00:48:26.815993 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.815913    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-metrics-certs\") pod \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") "
Sep 08 00:48:26.815993 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.815946    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hdsp\" (UniqueName: \"kubernetes.io/projected/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-kube-api-access-7hdsp\") pod \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\" (UID: \"fc7eadd3-eaee-492b-8b64-4fe6b3a36422\") "
Sep 08 00:48:26.816304 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.816277    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc7eadd3-eaee-492b-8b64-4fe6b3a36422" (UID: "fc7eadd3-eaee-492b-8b64-4fe6b3a36422"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:48:26.818170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.818139    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-kube-api-access-7hdsp" (OuterVolumeSpecName: "kube-api-access-7hdsp") pod "fc7eadd3-eaee-492b-8b64-4fe6b3a36422" (UID: "fc7eadd3-eaee-492b-8b64-4fe6b3a36422"). InnerVolumeSpecName "kube-api-access-7hdsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:48:26.818637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.818340    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc7eadd3-eaee-492b-8b64-4fe6b3a36422" (UID: "fc7eadd3-eaee-492b-8b64-4fe6b3a36422"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:48:26.818929 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.818904    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc7eadd3-eaee-492b-8b64-4fe6b3a36422" (UID: "fc7eadd3-eaee-492b-8b64-4fe6b3a36422"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:48:26.917473 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.917446    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:26.917473 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.917472    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdsp\" (UniqueName: \"kubernetes.io/projected/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-kube-api-access-7hdsp\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:26.917636 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.917486    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:26.917636 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:26.917498    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc7eadd3-eaee-492b-8b64-4fe6b3a36422-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:27.150581 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.150546    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:27.163250 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.163222    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:27.300653 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.300615    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc7eadd3-eaee-492b-8b64-4fe6b3a36422" containerID="fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71" exitCode=0
Sep 08 00:48:27.300846 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.300659    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"
Sep 08 00:48:27.300846 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.300696    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw" event={"ID":"fc7eadd3-eaee-492b-8b64-4fe6b3a36422","Type":"ContainerDied","Data":"fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71"}
Sep 08 00:48:27.300846 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.300751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw" event={"ID":"fc7eadd3-eaee-492b-8b64-4fe6b3a36422","Type":"ContainerDied","Data":"3823563a29da6c2af1d35fcfa2ce0beb5a1893ff500bbd54a906fcba7ec787bf"}
Sep 08 00:48:27.300846 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.300781    2555 scope.go:117] "RemoveContainer" containerID="fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71"
Sep 08 00:48:27.311880 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.311861    2555 scope.go:117] "RemoveContainer" containerID="fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71"
Sep 08 00:48:27.312199 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:48:27.312180    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71\": container with ID starting with fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71 not found: ID does not exist" containerID="fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71"
Sep 08 00:48:27.312260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.312207    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71"} err="failed to get container status \"fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71\": rpc error: code = NotFound desc = could not find container \"fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71\": container with ID starting with fe34e87d4bae571cdcead1ba0b25f72b0207a428278b82963919b4769df5da71 not found: ID does not exist"
Sep 08 00:48:27.316010 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.315982    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:27.338483 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.338457    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"]
Sep 08 00:48:27.351480 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:27.351445    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f627f164-57bdd847dd-pmvpw"]
Sep 08 00:48:28.155117 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:28.155075    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc7eadd3-eaee-492b-8b64-4fe6b3a36422" path="/var/lib/kubelet/pods/fc7eadd3-eaee-492b-8b64-4fe6b3a36422/volumes"
Sep 08 00:48:40.439374 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:40.439338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:48:41.875010 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:41.874977    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"]
Sep 08 00:48:56.896748 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:56.896690    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96" podUID="1753be71-bd43-47cf-9e2d-b32d151fac8c" containerName="main" containerID="cri-o://1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d" gracePeriod=45
Sep 08 00:48:57.163288 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.163215    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:57.213693 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.213658    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs6wn\" (UniqueName: \"kubernetes.io/projected/1753be71-bd43-47cf-9e2d-b32d151fac8c-kube-api-access-hs6wn\") pod \"1753be71-bd43-47cf-9e2d-b32d151fac8c\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") "
Sep 08 00:48:57.213904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.213719    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-tmp-dir\") pod \"1753be71-bd43-47cf-9e2d-b32d151fac8c\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") "
Sep 08 00:48:57.213904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.213756    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-dshm\") pod \"1753be71-bd43-47cf-9e2d-b32d151fac8c\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") "
Sep 08 00:48:57.213904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.213850    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753be71-bd43-47cf-9e2d-b32d151fac8c-tls-certs\") pod \"1753be71-bd43-47cf-9e2d-b32d151fac8c\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") "
Sep 08 00:48:57.214080 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.213917    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-model-cache\") pod \"1753be71-bd43-47cf-9e2d-b32d151fac8c\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") "
Sep 08 00:48:57.214080 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.213994    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-home\") pod \"1753be71-bd43-47cf-9e2d-b32d151fac8c\" (UID: \"1753be71-bd43-47cf-9e2d-b32d151fac8c\") "
Sep 08 00:48:57.214080 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.214021    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1753be71-bd43-47cf-9e2d-b32d151fac8c" (UID: "1753be71-bd43-47cf-9e2d-b32d151fac8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:48:57.214254 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.214199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1753be71-bd43-47cf-9e2d-b32d151fac8c" (UID: "1753be71-bd43-47cf-9e2d-b32d151fac8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:48:57.214325 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.214296    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-home" (OuterVolumeSpecName: "home") pod "1753be71-bd43-47cf-9e2d-b32d151fac8c" (UID: "1753be71-bd43-47cf-9e2d-b32d151fac8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:48:57.214450 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.214418    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:57.214502 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.214458    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:57.214502 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.214474    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:57.216070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.216048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-dshm" (OuterVolumeSpecName: "dshm") pod "1753be71-bd43-47cf-9e2d-b32d151fac8c" (UID: "1753be71-bd43-47cf-9e2d-b32d151fac8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:48:57.216169 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.216117    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1753be71-bd43-47cf-9e2d-b32d151fac8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1753be71-bd43-47cf-9e2d-b32d151fac8c" (UID: "1753be71-bd43-47cf-9e2d-b32d151fac8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:48:57.216221 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.216176    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1753be71-bd43-47cf-9e2d-b32d151fac8c-kube-api-access-hs6wn" (OuterVolumeSpecName: "kube-api-access-hs6wn") pod "1753be71-bd43-47cf-9e2d-b32d151fac8c" (UID: "1753be71-bd43-47cf-9e2d-b32d151fac8c"). InnerVolumeSpecName "kube-api-access-hs6wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:48:57.315123 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.315083    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6wn\" (UniqueName: \"kubernetes.io/projected/1753be71-bd43-47cf-9e2d-b32d151fac8c-kube-api-access-hs6wn\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:57.315123 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.315115    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753be71-bd43-47cf-9e2d-b32d151fac8c-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:57.315123 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.315124    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753be71-bd43-47cf-9e2d-b32d151fac8c-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:48:57.476551 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.476457    2555 generic.go:356] "Generic (PLEG): container finished" podID="1753be71-bd43-47cf-9e2d-b32d151fac8c" containerID="1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d" exitCode=0
Sep 08 00:48:57.476551 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.476509    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"
Sep 08 00:48:57.476551 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.476536    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96" event={"ID":"1753be71-bd43-47cf-9e2d-b32d151fac8c","Type":"ContainerDied","Data":"1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d"}
Sep 08 00:48:57.476832 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.476576    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96" event={"ID":"1753be71-bd43-47cf-9e2d-b32d151fac8c","Type":"ContainerDied","Data":"07c36f402efeaff0cf573766ebdedd77e970479c563bc62e4e5dd92a23894399"}
Sep 08 00:48:57.476832 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.476611    2555 scope.go:117] "RemoveContainer" containerID="1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d"
Sep 08 00:48:57.485853 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.485835    2555 scope.go:117] "RemoveContainer" containerID="1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d"
Sep 08 00:48:57.486111 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:48:57.486094    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d\": container with ID starting with 1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d not found: ID does not exist" containerID="1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d"
Sep 08 00:48:57.486159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.486123    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d"} err="failed to get container status \"1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d\": rpc error: code = NotFound desc = could not find container \"1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d\": container with ID starting with 1f1e2b9b744286f45b5a315354d4551bb54041668e7ec60ea42d282790a6747d not found: ID does not exist"
Sep 08 00:48:57.498587 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.498564    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"]
Sep 08 00:48:57.503708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:57.503689    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-e4902f/explicit-model-e4902f-kserve-68f5fc677d-7kq96"]
Sep 08 00:48:58.155394 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:48:58.155360    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1753be71-bd43-47cf-9e2d-b32d151fac8c" path="/var/lib/kubelet/pods/1753be71-bd43-47cf-9e2d-b32d151fac8c/volumes"
Sep 08 00:49:15.840710 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:15.840621    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:49:20.328128 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.328092    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"]
Sep 08 00:49:20.328542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.328526    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc7eadd3-eaee-492b-8b64-4fe6b3a36422" containerName="maas-api"
Sep 08 00:49:20.328604 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.328542    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc7eadd3-eaee-492b-8b64-4fe6b3a36422" containerName="maas-api"
Sep 08 00:49:20.328604 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.328560    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1753be71-bd43-47cf-9e2d-b32d151fac8c" containerName="main"
Sep 08 00:49:20.328604 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.328566    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1753be71-bd43-47cf-9e2d-b32d151fac8c" containerName="main"
Sep 08 00:49:20.328709 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.328675    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1753be71-bd43-47cf-9e2d-b32d151fac8c" containerName="main"
Sep 08 00:49:20.328709 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.328690    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc7eadd3-eaee-492b-8b64-4fe6b3a36422" containerName="maas-api"
Sep 08 00:49:20.331975 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.331953    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.334506 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.334481    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f04ef7-serving-cert\""
Sep 08 00:49:20.344508 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.344484    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"]
Sep 08 00:49:20.438248 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.438214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.438501 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.438480    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.438651 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.438632    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.438833 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.438817    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltcq\" (UniqueName: \"kubernetes.io/projected/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-kube-api-access-jltcq\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.539362 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.539332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltcq\" (UniqueName: \"kubernetes.io/projected/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-kube-api-access-jltcq\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.539546 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.539401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.539546 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.539435    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.539546 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.539456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.540236 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.540216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.541948 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.541921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.542054 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.541952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.547411 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.547389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltcq\" (UniqueName: \"kubernetes.io/projected/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-kube-api-access-jltcq\") pod \"maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.643471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.643409    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:20.915554 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.915527    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"]
Sep 08 00:49:20.917746 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:49:20.917718    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61dc3c05_6f53_43ad_bb96_a0e09dca0b23.slice/crio-634eabbb93313d75948837253b1196acee60b5bcd5880b7ca6b6c8fb9c824cdf WatchSource:0}: Error finding container 634eabbb93313d75948837253b1196acee60b5bcd5880b7ca6b6c8fb9c824cdf: Status 404 returned error can't find the container with id 634eabbb93313d75948837253b1196acee60b5bcd5880b7ca6b6c8fb9c824cdf
Sep 08 00:49:20.919473 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:20.919434    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:49:21.615905 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:21.615870    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5" event={"ID":"61dc3c05-6f53-43ad-bb96-a0e09dca0b23","Type":"ContainerStarted","Data":"805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d"}
Sep 08 00:49:21.616267 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:21.615928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5" event={"ID":"61dc3c05-6f53-43ad-bb96-a0e09dca0b23","Type":"ContainerStarted","Data":"634eabbb93313d75948837253b1196acee60b5bcd5880b7ca6b6c8fb9c824cdf"}
Sep 08 00:49:21.616267 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:21.616008    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:21.635309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:21.635211    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5" podStartSLOduration=1.176959596 podStartE2EDuration="1.635195219s" totalImagesPullingTime="458.235623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:49:21.634383271 +0000 UTC m=+995.977686293" watchObservedRunningTime="2026-09-08 00:49:21.635195219 +0000 UTC m=+995.978498237"
Sep 08 00:49:27.628108 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:27.628078    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:49:28.322085 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.322051    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"]
Sep 08 00:49:28.325903 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.325880    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.328573 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.328550    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-75c97b\"/\"noten-model-75c97b-kserve-self-signed-certs\""
Sep 08 00:49:28.328702 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.328551    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-75c97b\"/\"kube-root-ca.crt\""
Sep 08 00:49:28.329542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.329524    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-75c97b\"/\"openshift-service-ca.crt\""
Sep 08 00:49:28.329710 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.329689    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-75c97b\"/\"default-dockercfg-vjm8r\""
Sep 08 00:49:28.336285 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.336263    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"]
Sep 08 00:49:28.417199 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.417152    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22px\" (UniqueName: \"kubernetes.io/projected/6d391284-564b-4de9-b7a6-462c17c9d0ac-kube-api-access-x22px\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.417387 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.417209    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-home\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.417387 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.417294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d391284-564b-4de9-b7a6-462c17c9d0ac-tls-certs\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.417387 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.417329    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-dshm\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.417545 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.417400    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-model-cache\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.417545 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.417432    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-tmp-dir\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518067 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518032    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22px\" (UniqueName: \"kubernetes.io/projected/6d391284-564b-4de9-b7a6-462c17c9d0ac-kube-api-access-x22px\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-home\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d391284-564b-4de9-b7a6-462c17c9d0ac-tls-certs\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-dshm\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-model-cache\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518266 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518195    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-tmp-dir\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518719 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518690    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-tmp-dir\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518864 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518731    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-model-cache\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.518935 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.518862    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-home\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.520473 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.520438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-dshm\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.520961 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.520944    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d391284-564b-4de9-b7a6-462c17c9d0ac-tls-certs\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.528087 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.528059    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22px\" (UniqueName: \"kubernetes.io/projected/6d391284-564b-4de9-b7a6-462c17c9d0ac-kube-api-access-x22px\") pod \"noten-model-75c97b-kserve-75f5c66d48-r2z6t\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") " pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.636564 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.636490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:28.914031 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:28.914003    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"]
Sep 08 00:49:28.915886 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:49:28.915858    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d391284_564b_4de9_b7a6_462c17c9d0ac.slice/crio-c3d28436f9c6d3922d42782e14e576823f19d36cd489c68f157eef95437cb488 WatchSource:0}: Error finding container c3d28436f9c6d3922d42782e14e576823f19d36cd489c68f157eef95437cb488: Status 404 returned error can't find the container with id c3d28436f9c6d3922d42782e14e576823f19d36cd489c68f157eef95437cb488
Sep 08 00:49:29.662068 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:29.662029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t" event={"ID":"6d391284-564b-4de9-b7a6-462c17c9d0ac","Type":"ContainerStarted","Data":"5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25"}
Sep 08 00:49:29.662068 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:29.662071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t" event={"ID":"6d391284-564b-4de9-b7a6-462c17c9d0ac","Type":"ContainerStarted","Data":"c3d28436f9c6d3922d42782e14e576823f19d36cd489c68f157eef95437cb488"}
Sep 08 00:49:29.679631 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:29.679578    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t" podStartSLOduration=1.679561825 podStartE2EDuration="1.679561825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:49:29.678455879 +0000 UTC m=+1004.021758911" watchObservedRunningTime="2026-09-08 00:49:29.679561825 +0000 UTC m=+1004.022864844"
Sep 08 00:49:30.637216 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:30.637174    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:35.879084 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:35.879044    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"]
Sep 08 00:49:38.637082 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:38.637046    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:38.649647 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:38.649608    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:38.725414 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:38.725383    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:40.825325 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:40.825289    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"]
Sep 08 00:49:41.636438 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:41.636398    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:49:43.519485 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:43.519438    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"]
Sep 08 00:49:44.638519 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.638475    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-5872-139882422240720"]
Sep 08 00:49:44.642204 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.642181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:44.644673 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.644654    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fj6d7\""
Sep 08 00:49:44.648912 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.648888    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"]
Sep 08 00:49:44.774100 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.774066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmfw6\" (UniqueName: \"kubernetes.io/projected/6eb9a43c-12e5-4363-92d5-ac982afc7a33-kube-api-access-hmfw6\") pod \"test-curl-5872-139882422240720\" (UID: \"6eb9a43c-12e5-4363-92d5-ac982afc7a33\") " pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:44.874675 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.874640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmfw6\" (UniqueName: \"kubernetes.io/projected/6eb9a43c-12e5-4363-92d5-ac982afc7a33-kube-api-access-hmfw6\") pod \"test-curl-5872-139882422240720\" (UID: \"6eb9a43c-12e5-4363-92d5-ac982afc7a33\") " pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:44.882567 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.882538    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmfw6\" (UniqueName: \"kubernetes.io/projected/6eb9a43c-12e5-4363-92d5-ac982afc7a33-kube-api-access-hmfw6\") pod \"test-curl-5872-139882422240720\" (UID: \"6eb9a43c-12e5-4363-92d5-ac982afc7a33\") " pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:44.953657 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:44.953532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:45.227094 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:45.227060    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"]
Sep 08 00:49:45.236739 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:49:45.236706    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eb9a43c_12e5_4363_92d5_ac982afc7a33.slice/crio-e71ae46d21e04d9019a94c9e0549404d5aafa872fc6c8978db526e158c6bc38a WatchSource:0}: Error finding container e71ae46d21e04d9019a94c9e0549404d5aafa872fc6c8978db526e158c6bc38a: Status 404 returned error can't find the container with id e71ae46d21e04d9019a94c9e0549404d5aafa872fc6c8978db526e158c6bc38a
Sep 08 00:49:45.756474 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:45.756435    2555 generic.go:356] "Generic (PLEG): container finished" podID="6eb9a43c-12e5-4363-92d5-ac982afc7a33" containerID="ebd5af503dd387ff6dfb6aa99505ce67f0109fd6b1edaf3ef2f493157e444240" exitCode=0
Sep 08 00:49:45.756883 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:45.756521    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422240720" event={"ID":"6eb9a43c-12e5-4363-92d5-ac982afc7a33","Type":"ContainerDied","Data":"ebd5af503dd387ff6dfb6aa99505ce67f0109fd6b1edaf3ef2f493157e444240"}
Sep 08 00:49:45.756883 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:45.756568    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422240720" event={"ID":"6eb9a43c-12e5-4363-92d5-ac982afc7a33","Type":"ContainerStarted","Data":"e71ae46d21e04d9019a94c9e0549404d5aafa872fc6c8978db526e158c6bc38a"}
Sep 08 00:49:46.233876 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:46.233801    2555 scope.go:117] "RemoveContainer" containerID="9881d571057c0de8d9a47d2ed35d0dd2717e83c3daf025efad23660dc87dc02c"
Sep 08 00:49:46.904805 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:46.904778    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:46.937247 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:46.937224    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-5872-139882422240720_6eb9a43c-12e5-4363-92d5-ac982afc7a33/test-curl-5872-139882422240720/0.log"
Sep 08 00:49:46.960172 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:46.960147    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"]
Sep 08 00:49:46.963892 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:46.963871    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"]
Sep 08 00:49:46.995246 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:46.995222    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmfw6\" (UniqueName: \"kubernetes.io/projected/6eb9a43c-12e5-4363-92d5-ac982afc7a33-kube-api-access-hmfw6\") pod \"6eb9a43c-12e5-4363-92d5-ac982afc7a33\" (UID: \"6eb9a43c-12e5-4363-92d5-ac982afc7a33\") "
Sep 08 00:49:46.997111 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:46.997087    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb9a43c-12e5-4363-92d5-ac982afc7a33-kube-api-access-hmfw6" (OuterVolumeSpecName: "kube-api-access-hmfw6") pod "6eb9a43c-12e5-4363-92d5-ac982afc7a33" (UID: "6eb9a43c-12e5-4363-92d5-ac982afc7a33"). InnerVolumeSpecName "kube-api-access-hmfw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:49:47.096910 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:47.096885    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmfw6\" (UniqueName: \"kubernetes.io/projected/6eb9a43c-12e5-4363-92d5-ac982afc7a33-kube-api-access-hmfw6\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:47.769406 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:47.769373    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e71ae46d21e04d9019a94c9e0549404d5aafa872fc6c8978db526e158c6bc38a"
Sep 08 00:49:47.769406 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:47.769380    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:47.779301 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:49:47.779268    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-5872-139882422240720\" is forbidden: User \"system:node:ip-10-0-142-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-98.ec2.internal' and this object" podUID="6eb9a43c-12e5-4363-92d5-ac982afc7a33" pod="opendatahub/test-curl-5872-139882422240720"
Sep 08 00:49:48.153708 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:48.153678    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eb9a43c-12e5-4363-92d5-ac982afc7a33" path="/var/lib/kubelet/pods/6eb9a43c-12e5-4363-92d5-ac982afc7a33/volumes"
Sep 08 00:49:50.899952 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:50.899914    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl" podUID="3d06118b-e985-4c14-87c7-a9200342b34d" containerName="main" containerID="cri-o://045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1" gracePeriod=45
Sep 08 00:49:51.162970 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.162905    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:49:51.342885 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.342854    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-dshm\") pod \"3d06118b-e985-4c14-87c7-a9200342b34d\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") "
Sep 08 00:49:51.343070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.342894    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-tmp-dir\") pod \"3d06118b-e985-4c14-87c7-a9200342b34d\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") "
Sep 08 00:49:51.343070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.342917    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-home\") pod \"3d06118b-e985-4c14-87c7-a9200342b34d\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") "
Sep 08 00:49:51.343070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.342973    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k92tg\" (UniqueName: \"kubernetes.io/projected/3d06118b-e985-4c14-87c7-a9200342b34d-kube-api-access-k92tg\") pod \"3d06118b-e985-4c14-87c7-a9200342b34d\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") "
Sep 08 00:49:51.343070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343023    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d06118b-e985-4c14-87c7-a9200342b34d-tls-certs\") pod \"3d06118b-e985-4c14-87c7-a9200342b34d\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") "
Sep 08 00:49:51.343070 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343068    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-model-cache\") pod \"3d06118b-e985-4c14-87c7-a9200342b34d\" (UID: \"3d06118b-e985-4c14-87c7-a9200342b34d\") "
Sep 08 00:49:51.343342 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343114    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d06118b-e985-4c14-87c7-a9200342b34d" (UID: "3d06118b-e985-4c14-87c7-a9200342b34d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:51.343342 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343212    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-home" (OuterVolumeSpecName: "home") pod "3d06118b-e985-4c14-87c7-a9200342b34d" (UID: "3d06118b-e985-4c14-87c7-a9200342b34d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:51.343446 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343370    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d06118b-e985-4c14-87c7-a9200342b34d" (UID: "3d06118b-e985-4c14-87c7-a9200342b34d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:51.343844 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343541    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:51.343844 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343562    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:51.343844 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.343574    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:51.345178 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.345148    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-dshm" (OuterVolumeSpecName: "dshm") pod "3d06118b-e985-4c14-87c7-a9200342b34d" (UID: "3d06118b-e985-4c14-87c7-a9200342b34d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:51.345286 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.345195    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d06118b-e985-4c14-87c7-a9200342b34d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d06118b-e985-4c14-87c7-a9200342b34d" (UID: "3d06118b-e985-4c14-87c7-a9200342b34d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:49:51.345286 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.345249    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d06118b-e985-4c14-87c7-a9200342b34d-kube-api-access-k92tg" (OuterVolumeSpecName: "kube-api-access-k92tg") pod "3d06118b-e985-4c14-87c7-a9200342b34d" (UID: "3d06118b-e985-4c14-87c7-a9200342b34d"). InnerVolumeSpecName "kube-api-access-k92tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:49:51.445099 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.445005    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92tg\" (UniqueName: \"kubernetes.io/projected/3d06118b-e985-4c14-87c7-a9200342b34d-kube-api-access-k92tg\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:51.445099 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.445039    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d06118b-e985-4c14-87c7-a9200342b34d-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:51.445099 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.445050    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d06118b-e985-4c14-87c7-a9200342b34d-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:51.800481 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.800387    2555 generic.go:356] "Generic (PLEG): container finished" podID="3d06118b-e985-4c14-87c7-a9200342b34d" containerID="045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1" exitCode=0
Sep 08 00:49:51.800481 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.800435    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"
Sep 08 00:49:51.800481 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.800467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl" event={"ID":"3d06118b-e985-4c14-87c7-a9200342b34d","Type":"ContainerDied","Data":"045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1"}
Sep 08 00:49:51.800771 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.800513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl" event={"ID":"3d06118b-e985-4c14-87c7-a9200342b34d","Type":"ContainerDied","Data":"3c12b901d21a73c3f8475b5a390d2d9a54032ec9d200af3744ceb6c19aa10ff1"}
Sep 08 00:49:51.800771 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.800533    2555 scope.go:117] "RemoveContainer" containerID="045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1"
Sep 08 00:49:51.810319 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.810300    2555 scope.go:117] "RemoveContainer" containerID="045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1"
Sep 08 00:49:51.810622 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:49:51.810577    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1\": container with ID starting with 045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1 not found: ID does not exist" containerID="045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1"
Sep 08 00:49:51.810740 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.810631    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1"} err="failed to get container status \"045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1\": rpc error: code = NotFound desc = could not find container \"045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1\": container with ID starting with 045f0609bee381f594c4712188560ef6b308ddfa02282b635cbf324553b4e5b1 not found: ID does not exist"
Sep 08 00:49:51.821277 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.821253    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"]
Sep 08 00:49:51.825178 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:51.825153    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1cf84f/test-model-1cf84f-kserve-684564f7db-f8sjl"]
Sep 08 00:49:52.154921 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:52.154840    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d06118b-e985-4c14-87c7-a9200342b34d" path="/var/lib/kubelet/pods/3d06118b-e985-4c14-87c7-a9200342b34d/volumes"
Sep 08 00:49:54.461416 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.461381    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"]
Sep 08 00:49:54.461843 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.461583    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl" podUID="b0ab6739-cb46-4736-ac98-ada1f947a44a" containerName="payload-processing" containerID="cri-o://4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99" gracePeriod=30
Sep 08 00:49:54.726189 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.726123    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:49:54.773691 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.773661    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jpll\" (UniqueName: \"kubernetes.io/projected/b0ab6739-cb46-4736-ac98-ada1f947a44a-kube-api-access-4jpll\") pod \"b0ab6739-cb46-4736-ac98-ada1f947a44a\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") "
Sep 08 00:49:54.773896 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.773769    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab6739-cb46-4736-ac98-ada1f947a44a-plugins-config-volume\") pod \"b0ab6739-cb46-4736-ac98-ada1f947a44a\" (UID: \"b0ab6739-cb46-4736-ac98-ada1f947a44a\") "
Sep 08 00:49:54.774176 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.774149    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ab6739-cb46-4736-ac98-ada1f947a44a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0ab6739-cb46-4736-ac98-ada1f947a44a" (UID: "b0ab6739-cb46-4736-ac98-ada1f947a44a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:49:54.775814 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.775789    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ab6739-cb46-4736-ac98-ada1f947a44a-kube-api-access-4jpll" (OuterVolumeSpecName: "kube-api-access-4jpll") pod "b0ab6739-cb46-4736-ac98-ada1f947a44a" (UID: "b0ab6739-cb46-4736-ac98-ada1f947a44a"). InnerVolumeSpecName "kube-api-access-4jpll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:49:54.820992 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.820960    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0ab6739-cb46-4736-ac98-ada1f947a44a" containerID="4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99" exitCode=0
Sep 08 00:49:54.821171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.821021    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"
Sep 08 00:49:54.821171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.821047    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl" event={"ID":"b0ab6739-cb46-4736-ac98-ada1f947a44a","Type":"ContainerDied","Data":"4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99"}
Sep 08 00:49:54.821171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.821103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl" event={"ID":"b0ab6739-cb46-4736-ac98-ada1f947a44a","Type":"ContainerDied","Data":"a4bbcc3e5b8dfb040e65558924715188a02792ebca35ba4186ea5348ec48698e"}
Sep 08 00:49:54.821171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.821126    2555 scope.go:117] "RemoveContainer" containerID="4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99"
Sep 08 00:49:54.833124 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.833112    2555 scope.go:117] "RemoveContainer" containerID="4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99"
Sep 08 00:49:54.833823 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:49:54.833789    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99\": container with ID starting with 4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99 not found: ID does not exist" containerID="4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99"
Sep 08 00:49:54.833916 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.833825    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99"} err="failed to get container status \"4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99\": rpc error: code = NotFound desc = could not find container \"4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99\": container with ID starting with 4a98d5f199a91318973fe1c743db08a01a2958fff4d238fe9e5a3801ef236b99 not found: ID does not exist"
Sep 08 00:49:54.843679 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.843623    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"]
Sep 08 00:49:54.846665 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.846640    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1cf84f-7c4fd55ff7-hx7zl"]
Sep 08 00:49:54.875796 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.875766    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ab6739-cb46-4736-ac98-ada1f947a44a-plugins-config-volume\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:54.875796 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:54.875792    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jpll\" (UniqueName: \"kubernetes.io/projected/b0ab6739-cb46-4736-ac98-ada1f947a44a-kube-api-access-4jpll\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:55.846728 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:55.846689    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t" podUID="6d391284-564b-4de9-b7a6-462c17c9d0ac" containerName="main" containerID="cri-o://5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25" gracePeriod=45
Sep 08 00:49:56.093577 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.093549    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:56.154750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.154675    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ab6739-cb46-4736-ac98-ada1f947a44a" path="/var/lib/kubelet/pods/b0ab6739-cb46-4736-ac98-ada1f947a44a/volumes"
Sep 08 00:49:56.187757 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.187731    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-model-cache\") pod \"6d391284-564b-4de9-b7a6-462c17c9d0ac\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") "
Sep 08 00:49:56.187875 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.187776    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x22px\" (UniqueName: \"kubernetes.io/projected/6d391284-564b-4de9-b7a6-462c17c9d0ac-kube-api-access-x22px\") pod \"6d391284-564b-4de9-b7a6-462c17c9d0ac\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") "
Sep 08 00:49:56.187875 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.187800    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-dshm\") pod \"6d391284-564b-4de9-b7a6-462c17c9d0ac\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") "
Sep 08 00:49:56.187875 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.187866    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d391284-564b-4de9-b7a6-462c17c9d0ac-tls-certs\") pod \"6d391284-564b-4de9-b7a6-462c17c9d0ac\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") "
Sep 08 00:49:56.188035 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.187898    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-tmp-dir\") pod \"6d391284-564b-4de9-b7a6-462c17c9d0ac\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") "
Sep 08 00:49:56.188035 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.187936    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-home\") pod \"6d391284-564b-4de9-b7a6-462c17c9d0ac\" (UID: \"6d391284-564b-4de9-b7a6-462c17c9d0ac\") "
Sep 08 00:49:56.188035 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.188003    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d391284-564b-4de9-b7a6-462c17c9d0ac" (UID: "6d391284-564b-4de9-b7a6-462c17c9d0ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:56.188257 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.188217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d391284-564b-4de9-b7a6-462c17c9d0ac" (UID: "6d391284-564b-4de9-b7a6-462c17c9d0ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:56.188257 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.188229    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-home" (OuterVolumeSpecName: "home") pod "6d391284-564b-4de9-b7a6-462c17c9d0ac" (UID: "6d391284-564b-4de9-b7a6-462c17c9d0ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:56.188381 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.188295    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:56.188381 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.188311    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:56.188381 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.188324    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:56.190068 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.190046    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-dshm" (OuterVolumeSpecName: "dshm") pod "6d391284-564b-4de9-b7a6-462c17c9d0ac" (UID: "6d391284-564b-4de9-b7a6-462c17c9d0ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:56.190170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.190085    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d391284-564b-4de9-b7a6-462c17c9d0ac-kube-api-access-x22px" (OuterVolumeSpecName: "kube-api-access-x22px") pod "6d391284-564b-4de9-b7a6-462c17c9d0ac" (UID: "6d391284-564b-4de9-b7a6-462c17c9d0ac"). InnerVolumeSpecName "kube-api-access-x22px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:49:56.190170 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.190109    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d391284-564b-4de9-b7a6-462c17c9d0ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d391284-564b-4de9-b7a6-462c17c9d0ac" (UID: "6d391284-564b-4de9-b7a6-462c17c9d0ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:49:56.289583 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.289549    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x22px\" (UniqueName: \"kubernetes.io/projected/6d391284-564b-4de9-b7a6-462c17c9d0ac-kube-api-access-x22px\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:56.289583 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.289577    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d391284-564b-4de9-b7a6-462c17c9d0ac-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:56.289583 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.289586    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d391284-564b-4de9-b7a6-462c17c9d0ac-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:56.834357 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.834322    2555 generic.go:356] "Generic (PLEG): container finished" podID="6d391284-564b-4de9-b7a6-462c17c9d0ac" containerID="5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25" exitCode=0
Sep 08 00:49:56.834581 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.834381    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"
Sep 08 00:49:56.834581 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.834412    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t" event={"ID":"6d391284-564b-4de9-b7a6-462c17c9d0ac","Type":"ContainerDied","Data":"5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25"}
Sep 08 00:49:56.834581 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.834467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t" event={"ID":"6d391284-564b-4de9-b7a6-462c17c9d0ac","Type":"ContainerDied","Data":"c3d28436f9c6d3922d42782e14e576823f19d36cd489c68f157eef95437cb488"}
Sep 08 00:49:56.834581 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.834489    2555 scope.go:117] "RemoveContainer" containerID="5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25"
Sep 08 00:49:56.844547 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.844532    2555 scope.go:117] "RemoveContainer" containerID="5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25"
Sep 08 00:49:56.844816 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:49:56.844797    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25\": container with ID starting with 5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25 not found: ID does not exist" containerID="5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25"
Sep 08 00:49:56.844903 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.844821    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25"} err="failed to get container status \"5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25\": rpc error: code = NotFound desc = could not find container \"5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25\": container with ID starting with 5594afabebbd14f7c4ef534537d4cc51ae31258407c899d1e0ea47d5c63ebd25 not found: ID does not exist"
Sep 08 00:49:56.856698 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.856676    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"]
Sep 08 00:49:56.858613 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:56.858578    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-75c97b/noten-model-75c97b-kserve-75f5c66d48-r2z6t"]
Sep 08 00:49:58.154134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.154099    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d391284-564b-4de9-b7a6-462c17c9d0ac" path="/var/lib/kubelet/pods/6d391284-564b-4de9-b7a6-462c17c9d0ac/volumes"
Sep 08 00:49:58.541008 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.540918    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp" podUID="e47d5a24-685f-49c4-8d49-5da84e4cfe73" containerName="main" containerID="cri-o://b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4" gracePeriod=45
Sep 08 00:49:58.806663 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.806641    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:49:58.847759 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.847730    2555 generic.go:356] "Generic (PLEG): container finished" podID="e47d5a24-685f-49c4-8d49-5da84e4cfe73" containerID="b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4" exitCode=0
Sep 08 00:49:58.847759 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.847767    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"
Sep 08 00:49:58.847946 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.847813    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp" event={"ID":"e47d5a24-685f-49c4-8d49-5da84e4cfe73","Type":"ContainerDied","Data":"b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4"}
Sep 08 00:49:58.847946 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.847860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp" event={"ID":"e47d5a24-685f-49c4-8d49-5da84e4cfe73","Type":"ContainerDied","Data":"f5e45bedf038b89ca47a9c3bf63f49782deccdf7ce568206428c58eca1d6acfb"}
Sep 08 00:49:58.847946 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.847880    2555 scope.go:117] "RemoveContainer" containerID="b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4"
Sep 08 00:49:58.856973 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.856955    2555 scope.go:117] "RemoveContainer" containerID="b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4"
Sep 08 00:49:58.857262 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:49:58.857242    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4\": container with ID starting with b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4 not found: ID does not exist" containerID="b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4"
Sep 08 00:49:58.857338 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.857268    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4"} err="failed to get container status \"b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4\": rpc error: code = NotFound desc = could not find container \"b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4\": container with ID starting with b6b4e0fbf359d34e6fd76a5a15ab10d03130a8cdc47348d8934926ea44e9b8c4 not found: ID does not exist"
Sep 08 00:49:58.910365 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910341    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-home\") pod \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") "
Sep 08 00:49:58.910463 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910398    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tls-certs\") pod \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") "
Sep 08 00:49:58.910463 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910448    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tmp-dir\") pod \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") "
Sep 08 00:49:58.910570 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910479    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-dshm\") pod \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") "
Sep 08 00:49:58.910570 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910500    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-model-cache\") pod \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") "
Sep 08 00:49:58.910570 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910519    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zk7t\" (UniqueName: \"kubernetes.io/projected/e47d5a24-685f-49c4-8d49-5da84e4cfe73-kube-api-access-4zk7t\") pod \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\" (UID: \"e47d5a24-685f-49c4-8d49-5da84e4cfe73\") "
Sep 08 00:49:58.910750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910645    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-home" (OuterVolumeSpecName: "home") pod "e47d5a24-685f-49c4-8d49-5da84e4cfe73" (UID: "e47d5a24-685f-49c4-8d49-5da84e4cfe73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:58.910750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910728    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e47d5a24-685f-49c4-8d49-5da84e4cfe73" (UID: "e47d5a24-685f-49c4-8d49-5da84e4cfe73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:58.910862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910791    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-model-cache" (OuterVolumeSpecName: "model-cache") pod "e47d5a24-685f-49c4-8d49-5da84e4cfe73" (UID: "e47d5a24-685f-49c4-8d49-5da84e4cfe73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:58.910862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910832    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:58.910862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.910850    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:58.912625 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.912582    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-dshm" (OuterVolumeSpecName: "dshm") pod "e47d5a24-685f-49c4-8d49-5da84e4cfe73" (UID: "e47d5a24-685f-49c4-8d49-5da84e4cfe73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:49:58.912730 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.912681    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47d5a24-685f-49c4-8d49-5da84e4cfe73-kube-api-access-4zk7t" (OuterVolumeSpecName: "kube-api-access-4zk7t") pod "e47d5a24-685f-49c4-8d49-5da84e4cfe73" (UID: "e47d5a24-685f-49c4-8d49-5da84e4cfe73"). InnerVolumeSpecName "kube-api-access-4zk7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:49:58.912730 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:58.912689    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e47d5a24-685f-49c4-8d49-5da84e4cfe73" (UID: "e47d5a24-685f-49c4-8d49-5da84e4cfe73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:49:59.012259 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:59.012232    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zk7t\" (UniqueName: \"kubernetes.io/projected/e47d5a24-685f-49c4-8d49-5da84e4cfe73-kube-api-access-4zk7t\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:59.012259 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:59.012256    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47d5a24-685f-49c4-8d49-5da84e4cfe73-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:59.012405 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:59.012267    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:59.012405 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:59.012275    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47d5a24-685f-49c4-8d49-5da84e4cfe73-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:49:59.169517 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:59.169486    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"]
Sep 08 00:49:59.171342 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:49:59.171320    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e3896b/test-model-e3896b-kserve-bd876dd88-68dtp"]
Sep 08 00:50:00.160361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:00.160323    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e47d5a24-685f-49c4-8d49-5da84e4cfe73" path="/var/lib/kubelet/pods/e47d5a24-685f-49c4-8d49-5da84e4cfe73/volumes"
Sep 08 00:50:01.538854 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.538821    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"]
Sep 08 00:50:01.539394 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539368    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ab6739-cb46-4736-ac98-ada1f947a44a" containerName="payload-processing"
Sep 08 00:50:01.539394 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539395    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ab6739-cb46-4736-ac98-ada1f947a44a" containerName="payload-processing"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539407    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d391284-564b-4de9-b7a6-462c17c9d0ac" containerName="main"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539415    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d391284-564b-4de9-b7a6-462c17c9d0ac" containerName="main"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539434    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47d5a24-685f-49c4-8d49-5da84e4cfe73" containerName="main"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539442    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47d5a24-685f-49c4-8d49-5da84e4cfe73" containerName="main"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539451    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb9a43c-12e5-4363-92d5-ac982afc7a33" containerName="test-curl-5872-139882422240720"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539459    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb9a43c-12e5-4363-92d5-ac982afc7a33" containerName="test-curl-5872-139882422240720"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539493    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d06118b-e985-4c14-87c7-a9200342b34d" containerName="main"
Sep 08 00:50:01.539561 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539501    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d06118b-e985-4c14-87c7-a9200342b34d" containerName="main"
Sep 08 00:50:01.539907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539620    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d06118b-e985-4c14-87c7-a9200342b34d" containerName="main"
Sep 08 00:50:01.539907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539636    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e47d5a24-685f-49c4-8d49-5da84e4cfe73" containerName="main"
Sep 08 00:50:01.539907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539643    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d391284-564b-4de9-b7a6-462c17c9d0ac" containerName="main"
Sep 08 00:50:01.539907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539653    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ab6739-cb46-4736-ac98-ada1f947a44a" containerName="payload-processing"
Sep 08 00:50:01.539907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.539663    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb9a43c-12e5-4363-92d5-ac982afc7a33" containerName="test-curl-5872-139882422240720"
Sep 08 00:50:01.544578 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.544552    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.547199 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.547182    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c28b2155-serving-cert\""
Sep 08 00:50:01.556081 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.556056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"]
Sep 08 00:50:01.636497 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.636464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kj4l\" (UniqueName: \"kubernetes.io/projected/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-kube-api-access-6kj4l\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.636683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.636526    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.636683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.636577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-metrics-certs\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.636683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.636627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-openshift-service-ca\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.737984 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.737948    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kj4l\" (UniqueName: \"kubernetes.io/projected/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-kube-api-access-6kj4l\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.738196 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.738025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.738196 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.738064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-metrics-certs\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.738196 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.738099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-openshift-service-ca\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.738196 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:50:01.738158    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-serving-cert: secret "maas-api-e2e-mt-c28b2155-serving-cert" not found
Sep 08 00:50:01.738406 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:50:01.738262    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls podName:ea105caa-87fd-4941-84ab-8d0cbe5ebe96 nodeName:}" failed. No retries permitted until 2026-09-08 00:50:02.238242409 +0000 UTC m=+1036.581545428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls") pod "maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8" (UID: "ea105caa-87fd-4941-84ab-8d0cbe5ebe96") : secret "maas-api-e2e-mt-c28b2155-serving-cert" not found
Sep 08 00:50:01.738888 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.738864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-openshift-service-ca\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.740443 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.740416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-metrics-certs\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:01.748346 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:01.748327    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kj4l\" (UniqueName: \"kubernetes.io/projected/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-kube-api-access-6kj4l\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:02.243533 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:02.243501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:02.246342 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:02.246306    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls\") pod \"maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:02.456967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:02.456930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:02.731013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:02.730986    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"]
Sep 08 00:50:02.732577 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:50:02.732553    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea105caa_87fd_4941_84ab_8d0cbe5ebe96.slice/crio-28a84b58217f3114dd0dc8b53bcc4cf015138a66b5a636552b6ad5b4b43cbdd9 WatchSource:0}: Error finding container 28a84b58217f3114dd0dc8b53bcc4cf015138a66b5a636552b6ad5b4b43cbdd9: Status 404 returned error can't find the container with id 28a84b58217f3114dd0dc8b53bcc4cf015138a66b5a636552b6ad5b4b43cbdd9
Sep 08 00:50:02.875803 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:02.875756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8" event={"ID":"ea105caa-87fd-4941-84ab-8d0cbe5ebe96","Type":"ContainerStarted","Data":"28a84b58217f3114dd0dc8b53bcc4cf015138a66b5a636552b6ad5b4b43cbdd9"}
Sep 08 00:50:03.032046 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:03.032012    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:50:03.883833 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:03.883796    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8" event={"ID":"ea105caa-87fd-4941-84ab-8d0cbe5ebe96","Type":"ContainerStarted","Data":"5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f"}
Sep 08 00:50:03.884322 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:03.883923    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:03.902666 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:03.902564    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8" podStartSLOduration=2.407374416 podStartE2EDuration="2.902546839s" totalImagesPullingTime="495.172423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:03.901404008 +0000 UTC m=+1038.244707046" watchObservedRunningTime="2026-09-08 00:50:03.902546839 +0000 UTC m=+1038.245849858"
Sep 08 00:50:09.895260 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:09.895230    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:50:12.684075 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:12.684040    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"]
Sep 08 00:50:12.684516 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:12.684256    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr" podUID="0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" containerName="maas-api" containerID="cri-o://8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b" gracePeriod=30
Sep 08 00:50:12.717825 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:12.717787    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"]
Sep 08 00:50:12.721676 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:12.721648    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e3896b-29813805-6pb7b"]
Sep 08 00:50:13.140909 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.140883    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:50:13.241282 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.241189    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-maas-api-tls\") pod \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") "
Sep 08 00:50:13.241439 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.241282    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbbtn\" (UniqueName: \"kubernetes.io/projected/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-kube-api-access-hbbtn\") pod \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") "
Sep 08 00:50:13.241439 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.241355    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-metrics-certs\") pod \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") "
Sep 08 00:50:13.241439 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.241379    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-openshift-service-ca\") pod \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\" (UID: \"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec\") "
Sep 08 00:50:13.241868 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.241836    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" (UID: "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:50:13.243307 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.243280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" (UID: "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:50:13.243450 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.243433    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" (UID: "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:50:13.243491 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.243435    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-kube-api-access-hbbtn" (OuterVolumeSpecName: "kube-api-access-hbbtn") pod "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" (UID: "0400d54a-e6e1-483e-ab9a-7a2abfdce0ec"). InnerVolumeSpecName "kube-api-access-hbbtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:50:13.342745 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.342718    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:50:13.342745 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.342741    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:50:13.342745 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.342751    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:50:13.342926 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.342761    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbbtn\" (UniqueName: \"kubernetes.io/projected/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec-kube-api-access-hbbtn\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:50:13.942902 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.942866    2555 generic.go:356] "Generic (PLEG): container finished" podID="0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" containerID="8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b" exitCode=0
Sep 08 00:50:13.943279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.942915    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"
Sep 08 00:50:13.943279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.942948    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr" event={"ID":"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec","Type":"ContainerDied","Data":"8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b"}
Sep 08 00:50:13.943279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.943003    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr" event={"ID":"0400d54a-e6e1-483e-ab9a-7a2abfdce0ec","Type":"ContainerDied","Data":"71f3f3829c6d47104d2abc91b6a6192ddf8b7c30af43337c40bd190391e1bbe8"}
Sep 08 00:50:13.943279 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.943027    2555 scope.go:117] "RemoveContainer" containerID="8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b"
Sep 08 00:50:13.953444 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.953426    2555 scope.go:117] "RemoveContainer" containerID="8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b"
Sep 08 00:50:13.953725 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:50:13.953706    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b\": container with ID starting with 8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b not found: ID does not exist" containerID="8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b"
Sep 08 00:50:13.953777 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.953731    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b"} err="failed to get container status \"8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b\": rpc error: code = NotFound desc = could not find container \"8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b\": container with ID starting with 8f1e7ff486c529cbfd5f170fa7806386eb4f5c700acb2bd14395c4aa7972560b not found: ID does not exist"
Sep 08 00:50:13.991471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:13.991437    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"]
Sep 08 00:50:14.015917 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:14.015879    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e3896b-5d7c9d6b4-bm8pr"]
Sep 08 00:50:14.154544 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:14.154510    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" path="/var/lib/kubelet/pods/0400d54a-e6e1-483e-ab9a-7a2abfdce0ec/volumes"
Sep 08 00:50:14.154831 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:14.154817    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d" path="/var/lib/kubelet/pods/f7a72775-eb35-4e5d-a1e9-8d8f87c3ac1d/volumes"
Sep 08 00:50:21.234540 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:21.234502    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:50:24.632028 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.631992    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"]
Sep 08 00:50:24.632658 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.632635    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" containerName="maas-api"
Sep 08 00:50:24.632658 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.632660    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" containerName="maas-api"
Sep 08 00:50:24.632844 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.632789    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0400d54a-e6e1-483e-ab9a-7a2abfdce0ec" containerName="maas-api"
Sep 08 00:50:24.637487 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.637465    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.639916 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.639889    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b8c4de-serving-cert\""
Sep 08 00:50:24.648511 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.648485    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"]
Sep 08 00:50:24.748171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.748137    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwpsg\" (UniqueName: \"kubernetes.io/projected/b9279ece-eb66-4459-8396-85c1b519a07e-kube-api-access-bwpsg\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.748171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.748177    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9279ece-eb66-4459-8396-85c1b519a07e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.748393 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.748197    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.748393 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.748291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.849117 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.849087    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.849300 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.849154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwpsg\" (UniqueName: \"kubernetes.io/projected/b9279ece-eb66-4459-8396-85c1b519a07e-kube-api-access-bwpsg\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.849300 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.849180    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9279ece-eb66-4459-8396-85c1b519a07e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.849300 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.849196    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.849300 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:50:24.849256    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-serving-cert: secret "maas-api-e2e-shared-a-w3-b8c4de-serving-cert" not found
Sep 08 00:50:24.849522 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:50:24.849346    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls podName:b9279ece-eb66-4459-8396-85c1b519a07e nodeName:}" failed. No retries permitted until 2026-09-08 00:50:25.349324211 +0000 UTC m=+1059.692627208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls") pod "maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm" (UID: "b9279ece-eb66-4459-8396-85c1b519a07e") : secret "maas-api-e2e-shared-a-w3-b8c4de-serving-cert" not found
Sep 08 00:50:24.849974 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.849952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9279ece-eb66-4459-8396-85c1b519a07e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.851505 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.851479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:24.857197 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:24.857175    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwpsg\" (UniqueName: \"kubernetes.io/projected/b9279ece-eb66-4459-8396-85c1b519a07e-kube-api-access-bwpsg\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:25.353931 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:25.353897    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:25.356184 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:25.356158    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:25.549748 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:25.549708    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:25.822254 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:25.822228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"]
Sep 08 00:50:25.824331 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:50:25.824305    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9279ece_eb66_4459_8396_85c1b519a07e.slice/crio-7bb680b6fad66d296f7aeea02fa73e1e3419646909d6e9eabd8f4fe5a17505f6 WatchSource:0}: Error finding container 7bb680b6fad66d296f7aeea02fa73e1e3419646909d6e9eabd8f4fe5a17505f6: Status 404 returned error can't find the container with id 7bb680b6fad66d296f7aeea02fa73e1e3419646909d6e9eabd8f4fe5a17505f6
Sep 08 00:50:26.013113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:26.013070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm" event={"ID":"b9279ece-eb66-4459-8396-85c1b519a07e","Type":"ContainerStarted","Data":"7bb680b6fad66d296f7aeea02fa73e1e3419646909d6e9eabd8f4fe5a17505f6"}
Sep 08 00:50:27.020921 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:27.020885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm" event={"ID":"b9279ece-eb66-4459-8396-85c1b519a07e","Type":"ContainerStarted","Data":"23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4"}
Sep 08 00:50:27.021342 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:27.020963    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:27.037922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:27.037867    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm" podStartSLOduration=2.700058725 podStartE2EDuration="3.037850812s" totalImagesPullingTime="337.792087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:27.036487418 +0000 UTC m=+1061.379790437" watchObservedRunningTime="2026-09-08 00:50:27.037850812 +0000 UTC m=+1061.381153830"
Sep 08 00:50:33.032138 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:33.032102    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:50:40.739040 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:40.738959    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:50:53.221059 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.221010    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"]
Sep 08 00:50:53.226814 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.226784    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.229195 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.229167    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a8e810-serving-cert\""
Sep 08 00:50:53.233505 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.233474    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"]
Sep 08 00:50:53.307661 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.307627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.307827 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.307678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7ls\" (UniqueName: \"kubernetes.io/projected/83c44880-38cd-4b55-a75e-f4d52776a770-kube-api-access-5p7ls\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.307827 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.307745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44880-38cd-4b55-a75e-f4d52776a770-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.307827 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.307774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.408837 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.408807    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.409016 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.408855    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7ls\" (UniqueName: \"kubernetes.io/projected/83c44880-38cd-4b55-a75e-f4d52776a770-kube-api-access-5p7ls\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.409016 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.408907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44880-38cd-4b55-a75e-f4d52776a770-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.409016 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.408936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.409624 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.409579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44880-38cd-4b55-a75e-f4d52776a770-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.411148 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.411126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.411337 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.411316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.417258 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.417233    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7ls\" (UniqueName: \"kubernetes.io/projected/83c44880-38cd-4b55-a75e-f4d52776a770-kube-api-access-5p7ls\") pod \"maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.539977 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.539879    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:53.821499 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:53.821421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"]
Sep 08 00:50:53.822775 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:50:53.822749    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c44880_38cd_4b55_a75e_f4d52776a770.slice/crio-e69eaccae64a54776336b530617d0f2131bb8fb1dc79566b083a90ea56dadcfe WatchSource:0}: Error finding container e69eaccae64a54776336b530617d0f2131bb8fb1dc79566b083a90ea56dadcfe: Status 404 returned error can't find the container with id e69eaccae64a54776336b530617d0f2131bb8fb1dc79566b083a90ea56dadcfe
Sep 08 00:50:54.177483 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:54.177389    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7" event={"ID":"83c44880-38cd-4b55-a75e-f4d52776a770","Type":"ContainerStarted","Data":"e69eaccae64a54776336b530617d0f2131bb8fb1dc79566b083a90ea56dadcfe"}
Sep 08 00:50:55.184787 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.184752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7" event={"ID":"83c44880-38cd-4b55-a75e-f4d52776a770","Type":"ContainerStarted","Data":"f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e"}
Sep 08 00:50:55.185250 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.184808    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:50:55.207431 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.207359    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7" podStartSLOduration=1.814301519 podStartE2EDuration="2.207341473s" totalImagesPullingTime="393.039954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:55.205360058 +0000 UTC m=+1089.548663077" watchObservedRunningTime="2026-09-08 00:50:55.207341473 +0000 UTC m=+1089.550644484"
Sep 08 00:50:55.523533 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.523454    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"]
Sep 08 00:50:55.527166 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.527147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.529578 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.529545    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c28b2155\"/\"openshift-service-ca.crt\""
Sep 08 00:50:55.529578 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.529560    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c28b2155\"/\"default-dockercfg-x9srz\""
Sep 08 00:50:55.529729 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.529628    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c28b2155\"/\"kube-root-ca.crt\""
Sep 08 00:50:55.529779 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.529756    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c28b2155\"/\"e2e-lifecycle-model-c28b2155-kserve-self-signed-certs\""
Sep 08 00:50:55.538611 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.537712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"]
Sep 08 00:50:55.633152 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.633114    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqld\" (UniqueName: \"kubernetes.io/projected/21893528-7017-4df8-9737-d7cca71871a8-kube-api-access-ffqld\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.633333 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.633174    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-dshm\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.633333 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.633253    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-tmp-dir\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.633333 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.633295    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-home\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.633471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.633373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-model-cache\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.633471 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.633416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21893528-7017-4df8-9737-d7cca71871a8-tls-certs\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734436 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734384    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-model-cache\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734436 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21893528-7017-4df8-9737-d7cca71871a8-tls-certs\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734759 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734482    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqld\" (UniqueName: \"kubernetes.io/projected/21893528-7017-4df8-9737-d7cca71871a8-kube-api-access-ffqld\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734759 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-dshm\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734759 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-tmp-dir\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734759 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-home\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734920    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-model-cache\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.734976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.734950    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-tmp-dir\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.735050 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.735030    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-home\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.736792 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.736764    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-dshm\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.737073 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.737055    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21893528-7017-4df8-9737-d7cca71871a8-tls-certs\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.742134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.742112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqld\" (UniqueName: \"kubernetes.io/projected/21893528-7017-4df8-9737-d7cca71871a8-kube-api-access-ffqld\") pod \"e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") " pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:55.842078 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:55.842040    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:50:56.115919 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:56.115894    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"]
Sep 08 00:50:56.117341 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:50:56.117311    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21893528_7017_4df8_9737_d7cca71871a8.slice/crio-0c2f2735da9528178e5f2a3b2edfc5a01cf16bbc78a133230622a611c2940ed7 WatchSource:0}: Error finding container 0c2f2735da9528178e5f2a3b2edfc5a01cf16bbc78a133230622a611c2940ed7: Status 404 returned error can't find the container with id 0c2f2735da9528178e5f2a3b2edfc5a01cf16bbc78a133230622a611c2940ed7
Sep 08 00:50:56.191718 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:56.191688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n" event={"ID":"21893528-7017-4df8-9737-d7cca71871a8","Type":"ContainerStarted","Data":"d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc"}
Sep 08 00:50:56.192027 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:56.191737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n" event={"ID":"21893528-7017-4df8-9737-d7cca71871a8","Type":"ContainerStarted","Data":"0c2f2735da9528178e5f2a3b2edfc5a01cf16bbc78a133230622a611c2940ed7"}
Sep 08 00:50:56.208322 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:56.208273    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n" podStartSLOduration=1.208258068 podStartE2EDuration="1.208258068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:56.206641681 +0000 UTC m=+1090.549944702" watchObservedRunningTime="2026-09-08 00:50:56.208258068 +0000 UTC m=+1090.551561087"
Sep 08 00:50:56.842559 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:50:56.842524    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:51:01.197947 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:01.197918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:51:03.238094 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:03.238056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:51:05.843023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:05.842981    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:51:05.855637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:05.855608    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:51:06.259128 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:06.259051    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:51:33.831363 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:33.831323    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:51:35.131927 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.131884    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"]
Sep 08 00:51:35.144427 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.144402    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.145807 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.145780    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"]
Sep 08 00:51:35.148387 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.148367    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b8c4de\"/\"default-dockercfg-qtnq4\""
Sep 08 00:51:35.148697 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.148665    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b8c4de\"/\"kube-root-ca.crt\""
Sep 08 00:51:35.148808 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.148368    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b8c4de\"/\"openshift-service-ca.crt\""
Sep 08 00:51:35.148808 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.148534    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b8c4de\"/\"auth-test-model-b8c4de-kserve-self-signed-certs\""
Sep 08 00:51:35.215322 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.215290    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tmp-dir\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.215322 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.215327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-model-cache\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.215504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.215354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-home\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.215504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.215395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxwl\" (UniqueName: \"kubernetes.io/projected/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-kube-api-access-zkxwl\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.215504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.215469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tls-certs\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.215504 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.215491    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-dshm\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316424 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316385    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tmp-dir\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316424 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316422    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-model-cache\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316670 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-home\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316670 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxwl\" (UniqueName: \"kubernetes.io/projected/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-kube-api-access-zkxwl\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316749 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tls-certs\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316792 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316748    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-dshm\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316875 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316854    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-model-cache\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316931 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-home\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.316931 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.316915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tmp-dir\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.319135 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.319102    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-dshm\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.319346 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.319327    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tls-certs\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.324549 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.324522    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxwl\" (UniqueName: \"kubernetes.io/projected/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-kube-api-access-zkxwl\") pod \"auth-test-model-b8c4de-kserve-699c89b86d-bmbv9\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") " pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.457088 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.456981    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:35.728062 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:35.728036    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"]
Sep 08 00:51:35.729936 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:51:35.729908    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19bb7388_4f43_4238_b1ea_1ce8a05ffc0e.slice/crio-ee4916a340ecf5d97058d3068c41a2732c4e473022ee66228833a32d869f8b6e WatchSource:0}: Error finding container ee4916a340ecf5d97058d3068c41a2732c4e473022ee66228833a32d869f8b6e: Status 404 returned error can't find the container with id ee4916a340ecf5d97058d3068c41a2732c4e473022ee66228833a32d869f8b6e
Sep 08 00:51:36.421483 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:36.421438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9" event={"ID":"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e","Type":"ContainerStarted","Data":"7c75320da1203c3f936e91205d255e937273d4e0ecc1abb29f4b4fbea2c29433"}
Sep 08 00:51:36.421957 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:36.421494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9" event={"ID":"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e","Type":"ContainerStarted","Data":"ee4916a340ecf5d97058d3068c41a2732c4e473022ee66228833a32d869f8b6e"}
Sep 08 00:51:36.441649 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:36.441577    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9" podStartSLOduration=1.441558889 podStartE2EDuration="1.441558889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:51:36.438119916 +0000 UTC m=+1130.781423022" watchObservedRunningTime="2026-09-08 00:51:36.441558889 +0000 UTC m=+1130.784861913"
Sep 08 00:51:36.458014 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:36.457973    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:45.457916 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:45.457885    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:45.470441 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:45.470417    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:45.483871 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:45.483842    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:51:46.294812 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:46.294784    2555 scope.go:117] "RemoveContainer" containerID="caae96570c85ae87b7f4a9b31a0be362f09eff1fe6fa90fcb12872d4dcc78fbc"
Sep 08 00:51:46.303280 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:46.303263    2555 scope.go:117] "RemoveContainer" containerID="43930da827a2bf54665ffff491d94bff1599d00734b13b914af44a3c3db2ba37"
Sep 08 00:51:46.310470 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:46.310456    2555 scope.go:117] "RemoveContainer" containerID="881e883050b033f3bc96ec52017d480422824d23fe2eeed5a86cfa5863c7826b"
Sep 08 00:51:53.031327 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:53.031296    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:51:56.580255 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:56.580221    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"]
Sep 08 00:51:56.580763 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:56.580470    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8" podUID="ea105caa-87fd-4941-84ab-8d0cbe5ebe96" containerName="maas-api" containerID="cri-o://5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f" gracePeriod=30
Sep 08 00:51:56.849244 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:56.849221    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:51:57.023693 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.023656    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-metrics-certs\") pod \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") "
Sep 08 00:51:57.023881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.023732    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kj4l\" (UniqueName: \"kubernetes.io/projected/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-kube-api-access-6kj4l\") pod \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") "
Sep 08 00:51:57.023881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.023759    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls\") pod \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") "
Sep 08 00:51:57.023881 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.023831    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-openshift-service-ca\") pod \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\" (UID: \"ea105caa-87fd-4941-84ab-8d0cbe5ebe96\") "
Sep 08 00:51:57.024312 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.024277    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea105caa-87fd-4941-84ab-8d0cbe5ebe96" (UID: "ea105caa-87fd-4941-84ab-8d0cbe5ebe96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:51:57.025938 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.025905    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea105caa-87fd-4941-84ab-8d0cbe5ebe96" (UID: "ea105caa-87fd-4941-84ab-8d0cbe5ebe96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:51:57.026029 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.025983    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-kube-api-access-6kj4l" (OuterVolumeSpecName: "kube-api-access-6kj4l") pod "ea105caa-87fd-4941-84ab-8d0cbe5ebe96" (UID: "ea105caa-87fd-4941-84ab-8d0cbe5ebe96"). InnerVolumeSpecName "kube-api-access-6kj4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:51:57.026029 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.025994    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea105caa-87fd-4941-84ab-8d0cbe5ebe96" (UID: "ea105caa-87fd-4941-84ab-8d0cbe5ebe96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:51:57.125426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.125342    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:51:57.125426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.125373    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:51:57.125426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.125386    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kj4l\" (UniqueName: \"kubernetes.io/projected/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-kube-api-access-6kj4l\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:51:57.125426 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.125399    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea105caa-87fd-4941-84ab-8d0cbe5ebe96-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:51:57.541458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.541363    2555 generic.go:356] "Generic (PLEG): container finished" podID="ea105caa-87fd-4941-84ab-8d0cbe5ebe96" containerID="5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f" exitCode=0
Sep 08 00:51:57.541458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.541409    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8" event={"ID":"ea105caa-87fd-4941-84ab-8d0cbe5ebe96","Type":"ContainerDied","Data":"5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f"}
Sep 08 00:51:57.541458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.541417    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"
Sep 08 00:51:57.541458 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.541449    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8" event={"ID":"ea105caa-87fd-4941-84ab-8d0cbe5ebe96","Type":"ContainerDied","Data":"28a84b58217f3114dd0dc8b53bcc4cf015138a66b5a636552b6ad5b4b43cbdd9"}
Sep 08 00:51:57.541819 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.541470    2555 scope.go:117] "RemoveContainer" containerID="5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f"
Sep 08 00:51:57.551182 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.551164    2555 scope.go:117] "RemoveContainer" containerID="5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f"
Sep 08 00:51:57.551438 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:51:57.551422    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f\": container with ID starting with 5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f not found: ID does not exist" containerID="5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f"
Sep 08 00:51:57.551489 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.551445    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f"} err="failed to get container status \"5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f\": rpc error: code = NotFound desc = could not find container \"5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f\": container with ID starting with 5a087236f88873b30b3f2dff90f89e62ab295d252b657c53cbcb0a2f8a15c03f not found: ID does not exist"
Sep 08 00:51:57.562487 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.562464    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"]
Sep 08 00:51:57.566108 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:57.566087    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c28b2155-5c5c59bcdd-vvqn8"]
Sep 08 00:51:58.153192 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:51:58.153157    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea105caa-87fd-4941-84ab-8d0cbe5ebe96" path="/var/lib/kubelet/pods/ea105caa-87fd-4941-84ab-8d0cbe5ebe96/volumes"
Sep 08 00:52:04.731544 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:04.731505    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"]
Sep 08 00:52:15.030355 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:15.030271    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:52:19.751196 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:19.751136    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n" podUID="21893528-7017-4df8-9737-d7cca71871a8" containerName="main" containerID="cri-o://d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc" gracePeriod=45
Sep 08 00:52:20.009907 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.009843    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:52:20.139445 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139415    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-tmp-dir\") pod \"21893528-7017-4df8-9737-d7cca71871a8\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") "
Sep 08 00:52:20.139637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139455    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21893528-7017-4df8-9737-d7cca71871a8-tls-certs\") pod \"21893528-7017-4df8-9737-d7cca71871a8\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") "
Sep 08 00:52:20.139637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139544    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffqld\" (UniqueName: \"kubernetes.io/projected/21893528-7017-4df8-9737-d7cca71871a8-kube-api-access-ffqld\") pod \"21893528-7017-4df8-9737-d7cca71871a8\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") "
Sep 08 00:52:20.139637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139630    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-model-cache\") pod \"21893528-7017-4df8-9737-d7cca71871a8\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") "
Sep 08 00:52:20.139785 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139673    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-home\") pod \"21893528-7017-4df8-9737-d7cca71871a8\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") "
Sep 08 00:52:20.139785 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139732    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-dshm\") pod \"21893528-7017-4df8-9737-d7cca71871a8\" (UID: \"21893528-7017-4df8-9737-d7cca71871a8\") "
Sep 08 00:52:20.139785 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139740    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21893528-7017-4df8-9737-d7cca71871a8" (UID: "21893528-7017-4df8-9737-d7cca71871a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:52:20.139968 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139907    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "21893528-7017-4df8-9737-d7cca71871a8" (UID: "21893528-7017-4df8-9737-d7cca71871a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:52:20.139968 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.139923    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-home" (OuterVolumeSpecName: "home") pod "21893528-7017-4df8-9737-d7cca71871a8" (UID: "21893528-7017-4df8-9737-d7cca71871a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:52:20.140086 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.140060    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:52:20.140086 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.140074    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:52:20.140178 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.140087    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:52:20.141853 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.141828    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21893528-7017-4df8-9737-d7cca71871a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21893528-7017-4df8-9737-d7cca71871a8" (UID: "21893528-7017-4df8-9737-d7cca71871a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:52:20.141945 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.141831    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-dshm" (OuterVolumeSpecName: "dshm") pod "21893528-7017-4df8-9737-d7cca71871a8" (UID: "21893528-7017-4df8-9737-d7cca71871a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:52:20.141945 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.141892    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21893528-7017-4df8-9737-d7cca71871a8-kube-api-access-ffqld" (OuterVolumeSpecName: "kube-api-access-ffqld") pod "21893528-7017-4df8-9737-d7cca71871a8" (UID: "21893528-7017-4df8-9737-d7cca71871a8"). InnerVolumeSpecName "kube-api-access-ffqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:52:20.241308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.241281    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21893528-7017-4df8-9737-d7cca71871a8-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:52:20.241308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.241304    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21893528-7017-4df8-9737-d7cca71871a8-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:52:20.241308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.241314    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqld\" (UniqueName: \"kubernetes.io/projected/21893528-7017-4df8-9737-d7cca71871a8-kube-api-access-ffqld\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:52:20.674911 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.674872    2555 generic.go:356] "Generic (PLEG): container finished" podID="21893528-7017-4df8-9737-d7cca71871a8" containerID="d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc" exitCode=0
Sep 08 00:52:20.674911 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.674914    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"
Sep 08 00:52:20.675161 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.674949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n" event={"ID":"21893528-7017-4df8-9737-d7cca71871a8","Type":"ContainerDied","Data":"d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc"}
Sep 08 00:52:20.675161 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.674987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n" event={"ID":"21893528-7017-4df8-9737-d7cca71871a8","Type":"ContainerDied","Data":"0c2f2735da9528178e5f2a3b2edfc5a01cf16bbc78a133230622a611c2940ed7"}
Sep 08 00:52:20.675161 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.675008    2555 scope.go:117] "RemoveContainer" containerID="d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc"
Sep 08 00:52:20.685569 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.685552    2555 scope.go:117] "RemoveContainer" containerID="d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc"
Sep 08 00:52:20.685857 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:52:20.685839    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc\": container with ID starting with d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc not found: ID does not exist" containerID="d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc"
Sep 08 00:52:20.685919 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.685865    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc"} err="failed to get container status \"d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc\": rpc error: code = NotFound desc = could not find container \"d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc\": container with ID starting with d3253875b4b6543f6c6925066ef9adc8e99f3cf0bee15b6c4cb6568fe95e2ebc not found: ID does not exist"
Sep 08 00:52:20.693758 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.693721    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"]
Sep 08 00:52:20.697019 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:20.696996    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c28b2155/e2e-lifecycle-model-c28b2155-kserve-7b49c49cfb-zwg8n"]
Sep 08 00:52:22.153264 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:22.153231    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21893528-7017-4df8-9737-d7cca71871a8" path="/var/lib/kubelet/pods/21893528-7017-4df8-9737-d7cca71871a8/volumes"
Sep 08 00:52:46.189747 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:46.189715    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:52:46.192060 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:46.192035    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:52:55.529486 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:52:55.529448    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:53:11.432939 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:11.432899    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:53:45.328168 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.328095    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"]
Sep 08 00:53:45.328922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.328887    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21893528-7017-4df8-9737-d7cca71871a8" containerName="main"
Sep 08 00:53:45.328922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.328913    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="21893528-7017-4df8-9737-d7cca71871a8" containerName="main"
Sep 08 00:53:45.328922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.328927    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea105caa-87fd-4941-84ab-8d0cbe5ebe96" containerName="maas-api"
Sep 08 00:53:45.328922 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.328937    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea105caa-87fd-4941-84ab-8d0cbe5ebe96" containerName="maas-api"
Sep 08 00:53:45.329254 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.329043    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea105caa-87fd-4941-84ab-8d0cbe5ebe96" containerName="maas-api"
Sep 08 00:53:45.329254 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.329053    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="21893528-7017-4df8-9737-d7cca71871a8" containerName="main"
Sep 08 00:53:45.332189 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.332168    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.334749 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.334725    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-55b1e9-serving-cert\""
Sep 08 00:53:45.344474 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.344420    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"]
Sep 08 00:53:45.420998 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.420960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e040a327-067b-4512-a6b3-1aa142f8546a-openshift-service-ca\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.420998 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.420997    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcfs\" (UniqueName: \"kubernetes.io/projected/e040a327-067b-4512-a6b3-1aa142f8546a-kube-api-access-hkcfs\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.421251 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.421064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-metrics-certs\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.421251 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.421149    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.522395 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.522359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e040a327-067b-4512-a6b3-1aa142f8546a-openshift-service-ca\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.522395 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.522398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcfs\" (UniqueName: \"kubernetes.io/projected/e040a327-067b-4512-a6b3-1aa142f8546a-kube-api-access-hkcfs\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.522683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.522469    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-metrics-certs\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.522683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.522516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.522683 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:53:45.522648    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-serving-cert: secret "maas-api-e2e-api-a-55b1e9-serving-cert" not found
Sep 08 00:53:45.522842 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:53:45.522719    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls podName:e040a327-067b-4512-a6b3-1aa142f8546a nodeName:}" failed. No retries permitted until 2026-09-08 00:53:46.022697092 +0000 UTC m=+1260.366000089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls") pod "maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd" (UID: "e040a327-067b-4512-a6b3-1aa142f8546a") : secret "maas-api-e2e-api-a-55b1e9-serving-cert" not found
Sep 08 00:53:45.523030 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.523009    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e040a327-067b-4512-a6b3-1aa142f8546a-openshift-service-ca\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.524899 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.524879    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-metrics-certs\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:45.532794 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:45.532772    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcfs\" (UniqueName: \"kubernetes.io/projected/e040a327-067b-4512-a6b3-1aa142f8546a-kube-api-access-hkcfs\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:46.027060 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:46.027028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:46.029529 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:46.029502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls\") pod \"maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:46.244668 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:46.244637    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:46.516374 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:46.516347    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"]
Sep 08 00:53:46.518471 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:53:46.518444    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode040a327_067b_4512_a6b3_1aa142f8546a.slice/crio-ef40b774c5b6871eef3c0f8319e50b4906b1aa6b22dafa597f75f09a1819c787 WatchSource:0}: Error finding container ef40b774c5b6871eef3c0f8319e50b4906b1aa6b22dafa597f75f09a1819c787: Status 404 returned error can't find the container with id ef40b774c5b6871eef3c0f8319e50b4906b1aa6b22dafa597f75f09a1819c787
Sep 08 00:53:47.153526 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:47.153492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd" event={"ID":"e040a327-067b-4512-a6b3-1aa142f8546a","Type":"ContainerStarted","Data":"54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd"}
Sep 08 00:53:47.153715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:47.153542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd" event={"ID":"e040a327-067b-4512-a6b3-1aa142f8546a","Type":"ContainerStarted","Data":"ef40b774c5b6871eef3c0f8319e50b4906b1aa6b22dafa597f75f09a1819c787"}
Sep 08 00:53:47.153715 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:47.153638    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:53:47.173084 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:47.173030    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd" podStartSLOduration=1.7405265970000001 podStartE2EDuration="2.173015003s" totalImagesPullingTime="432.488406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:53:47.171415902 +0000 UTC m=+1261.514718922" watchObservedRunningTime="2026-09-08 00:53:47.173015003 +0000 UTC m=+1261.516318091"
Sep 08 00:53:51.232666 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:51.232619    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:53:53.164417 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:53:53.164386    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:54:14.535043 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.535002    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"]
Sep 08 00:54:14.538873 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.538852    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.541728 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.541703    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3e8800-serving-cert\""
Sep 08 00:54:14.549289 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.549260    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"]
Sep 08 00:54:14.612319 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.612276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc8pf\" (UniqueName: \"kubernetes.io/projected/af39042e-732d-4541-9af7-ee4ef35cb506-kube-api-access-wc8pf\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.612511 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.612391    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af39042e-732d-4541-9af7-ee4ef35cb506-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.612511 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.612442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.612654 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.612515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-metrics-certs\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.713669 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.713628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8pf\" (UniqueName: \"kubernetes.io/projected/af39042e-732d-4541-9af7-ee4ef35cb506-kube-api-access-wc8pf\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.713880 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.713716    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af39042e-732d-4541-9af7-ee4ef35cb506-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.713880 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.713746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.713880 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.713798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-metrics-certs\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.714043 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:54:14.713926    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-serving-cert: secret "maas-api-e2e-api-b-3e8800-serving-cert" not found
Sep 08 00:54:14.714043 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:54:14.713996    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls podName:af39042e-732d-4541-9af7-ee4ef35cb506 nodeName:}" failed. No retries permitted until 2026-09-08 00:54:15.213975005 +0000 UTC m=+1289.557278001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls") pod "maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w" (UID: "af39042e-732d-4541-9af7-ee4ef35cb506") : secret "maas-api-e2e-api-b-3e8800-serving-cert" not found
Sep 08 00:54:14.714363 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.714346    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af39042e-732d-4541-9af7-ee4ef35cb506-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.716192 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.716169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-metrics-certs\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:14.721884 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:14.721861    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8pf\" (UniqueName: \"kubernetes.io/projected/af39042e-732d-4541-9af7-ee4ef35cb506-kube-api-access-wc8pf\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:15.218011 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:15.217972    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:15.220444 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:15.220414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls\") pod \"maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:15.451156 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:15.451124    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:15.721786 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:15.721755    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"]
Sep 08 00:54:15.724031 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:54:15.724001    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf39042e_732d_4541_9af7_ee4ef35cb506.slice/crio-86f9dafb7bf3ba5fac422bef9a3e0ad9e7740f734d3444a8b6e651dfd6ef27b0 WatchSource:0}: Error finding container 86f9dafb7bf3ba5fac422bef9a3e0ad9e7740f734d3444a8b6e651dfd6ef27b0: Status 404 returned error can't find the container with id 86f9dafb7bf3ba5fac422bef9a3e0ad9e7740f734d3444a8b6e651dfd6ef27b0
Sep 08 00:54:16.311171 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:16.311129    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w" event={"ID":"af39042e-732d-4541-9af7-ee4ef35cb506","Type":"ContainerStarted","Data":"d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef"}
Sep 08 00:54:16.311429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:16.311187    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w" event={"ID":"af39042e-732d-4541-9af7-ee4ef35cb506","Type":"ContainerStarted","Data":"86f9dafb7bf3ba5fac422bef9a3e0ad9e7740f734d3444a8b6e651dfd6ef27b0"}
Sep 08 00:54:16.311429 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:16.311251    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:16.331092 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:16.331034    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w" podStartSLOduration=1.943730275 podStartE2EDuration="2.331013168s" totalImagesPullingTime="387.282893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:54:16.328340656 +0000 UTC m=+1290.671643676" watchObservedRunningTime="2026-09-08 00:54:16.331013168 +0000 UTC m=+1290.674316188"
Sep 08 00:54:22.322104 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:22.322072    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:54:26.335113 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:26.335077    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:54:53.878109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:53.877965    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"]
Sep 08 00:54:53.878749 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:53.878202    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm" podUID="b9279ece-eb66-4459-8396-85c1b519a07e" containerName="maas-api" containerID="cri-o://23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4" gracePeriod=30
Sep 08 00:54:54.139899 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.139837    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:54:54.194409 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.194377    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9279ece-eb66-4459-8396-85c1b519a07e-openshift-service-ca\") pod \"b9279ece-eb66-4459-8396-85c1b519a07e\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") "
Sep 08 00:54:54.194409 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.194411    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwpsg\" (UniqueName: \"kubernetes.io/projected/b9279ece-eb66-4459-8396-85c1b519a07e-kube-api-access-bwpsg\") pod \"b9279ece-eb66-4459-8396-85c1b519a07e\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") "
Sep 08 00:54:54.194642 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.194447    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-metrics-certs\") pod \"b9279ece-eb66-4459-8396-85c1b519a07e\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") "
Sep 08 00:54:54.194642 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.194468    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls\") pod \"b9279ece-eb66-4459-8396-85c1b519a07e\" (UID: \"b9279ece-eb66-4459-8396-85c1b519a07e\") "
Sep 08 00:54:54.194797 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.194765    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9279ece-eb66-4459-8396-85c1b519a07e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9279ece-eb66-4459-8396-85c1b519a07e" (UID: "b9279ece-eb66-4459-8396-85c1b519a07e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:54:54.196555 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.196518    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9279ece-eb66-4459-8396-85c1b519a07e" (UID: "b9279ece-eb66-4459-8396-85c1b519a07e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:54:54.196661 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.196534    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9279ece-eb66-4459-8396-85c1b519a07e" (UID: "b9279ece-eb66-4459-8396-85c1b519a07e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:54:54.196661 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.196643    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9279ece-eb66-4459-8396-85c1b519a07e-kube-api-access-bwpsg" (OuterVolumeSpecName: "kube-api-access-bwpsg") pod "b9279ece-eb66-4459-8396-85c1b519a07e" (UID: "b9279ece-eb66-4459-8396-85c1b519a07e"). InnerVolumeSpecName "kube-api-access-bwpsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:54:54.296097 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.296062    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9279ece-eb66-4459-8396-85c1b519a07e-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:54.296097 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.296095    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwpsg\" (UniqueName: \"kubernetes.io/projected/b9279ece-eb66-4459-8396-85c1b519a07e-kube-api-access-bwpsg\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:54.296308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.296108    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:54.296308 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.296120    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9279ece-eb66-4459-8396-85c1b519a07e-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:54.533519 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.533426    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9279ece-eb66-4459-8396-85c1b519a07e" containerID="23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4" exitCode=0
Sep 08 00:54:54.533519 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.533478    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"
Sep 08 00:54:54.533519 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.533512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm" event={"ID":"b9279ece-eb66-4459-8396-85c1b519a07e","Type":"ContainerDied","Data":"23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4"}
Sep 08 00:54:54.533787 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.533563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm" event={"ID":"b9279ece-eb66-4459-8396-85c1b519a07e","Type":"ContainerDied","Data":"7bb680b6fad66d296f7aeea02fa73e1e3419646909d6e9eabd8f4fe5a17505f6"}
Sep 08 00:54:54.533787 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.533586    2555 scope.go:117] "RemoveContainer" containerID="23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4"
Sep 08 00:54:54.543584 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.543566    2555 scope.go:117] "RemoveContainer" containerID="23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4"
Sep 08 00:54:54.543873 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:54:54.543853    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4\": container with ID starting with 23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4 not found: ID does not exist" containerID="23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4"
Sep 08 00:54:54.543933 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.543882    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4"} err="failed to get container status \"23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4\": rpc error: code = NotFound desc = could not find container \"23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4\": container with ID starting with 23ea2b2b44616142d5cd6351bdead9c1aae29fd36561ad5c2942245cbc86c5c4 not found: ID does not exist"
Sep 08 00:54:54.555537 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.555507    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"]
Sep 08 00:54:54.560287 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:54.560267    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b8c4de-558c489c5f-nrwmm"]
Sep 08 00:54:56.153442 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:56.153409    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9279ece-eb66-4459-8396-85c1b519a07e" path="/var/lib/kubelet/pods/b9279ece-eb66-4459-8396-85c1b519a07e/volumes"
Sep 08 00:54:56.742556 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:56.742521    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"]
Sep 08 00:54:56.742840 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:56.742814    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd" podUID="e040a327-067b-4512-a6b3-1aa142f8546a" containerName="maas-api" containerID="cri-o://54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd" gracePeriod=30
Sep 08 00:54:56.995478 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:56.995417    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:54:57.022467 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.022435    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e040a327-067b-4512-a6b3-1aa142f8546a-openshift-service-ca\") pod \"e040a327-067b-4512-a6b3-1aa142f8546a\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") "
Sep 08 00:54:57.022607 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.022527    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkcfs\" (UniqueName: \"kubernetes.io/projected/e040a327-067b-4512-a6b3-1aa142f8546a-kube-api-access-hkcfs\") pod \"e040a327-067b-4512-a6b3-1aa142f8546a\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") "
Sep 08 00:54:57.022607 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.022555    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-metrics-certs\") pod \"e040a327-067b-4512-a6b3-1aa142f8546a\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") "
Sep 08 00:54:57.022703 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.022643    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls\") pod \"e040a327-067b-4512-a6b3-1aa142f8546a\" (UID: \"e040a327-067b-4512-a6b3-1aa142f8546a\") "
Sep 08 00:54:57.022818 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.022793    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e040a327-067b-4512-a6b3-1aa142f8546a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e040a327-067b-4512-a6b3-1aa142f8546a" (UID: "e040a327-067b-4512-a6b3-1aa142f8546a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:54:57.023033 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.023012    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e040a327-067b-4512-a6b3-1aa142f8546a-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:57.024683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.024653    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e040a327-067b-4512-a6b3-1aa142f8546a" (UID: "e040a327-067b-4512-a6b3-1aa142f8546a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:54:57.024796 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.024765    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e040a327-067b-4512-a6b3-1aa142f8546a" (UID: "e040a327-067b-4512-a6b3-1aa142f8546a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:54:57.024925 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.024908    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e040a327-067b-4512-a6b3-1aa142f8546a-kube-api-access-hkcfs" (OuterVolumeSpecName: "kube-api-access-hkcfs") pod "e040a327-067b-4512-a6b3-1aa142f8546a" (UID: "e040a327-067b-4512-a6b3-1aa142f8546a"). InnerVolumeSpecName "kube-api-access-hkcfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:54:57.123789 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.123752    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcfs\" (UniqueName: \"kubernetes.io/projected/e040a327-067b-4512-a6b3-1aa142f8546a-kube-api-access-hkcfs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:57.123789 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.123782    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:57.123789 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.123791    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e040a327-067b-4512-a6b3-1aa142f8546a-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:54:57.553119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.553080    2555 generic.go:356] "Generic (PLEG): container finished" podID="e040a327-067b-4512-a6b3-1aa142f8546a" containerID="54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd" exitCode=0
Sep 08 00:54:57.553559 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.553128    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"
Sep 08 00:54:57.553559 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.553160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd" event={"ID":"e040a327-067b-4512-a6b3-1aa142f8546a","Type":"ContainerDied","Data":"54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd"}
Sep 08 00:54:57.553559 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.553212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd" event={"ID":"e040a327-067b-4512-a6b3-1aa142f8546a","Type":"ContainerDied","Data":"ef40b774c5b6871eef3c0f8319e50b4906b1aa6b22dafa597f75f09a1819c787"}
Sep 08 00:54:57.553559 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.553234    2555 scope.go:117] "RemoveContainer" containerID="54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd"
Sep 08 00:54:57.562643 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.562627    2555 scope.go:117] "RemoveContainer" containerID="54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd"
Sep 08 00:54:57.562898 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:54:57.562880    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd\": container with ID starting with 54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd not found: ID does not exist" containerID="54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd"
Sep 08 00:54:57.562957 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.562907    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd"} err="failed to get container status \"54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd\": rpc error: code = NotFound desc = could not find container \"54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd\": container with ID starting with 54e480e70d36b820d0012f0a01035089331a1e6fd1e8d52fc2a609d62e539cdd not found: ID does not exist"
Sep 08 00:54:57.575460 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.575436    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"]
Sep 08 00:54:57.579727 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:57.579707    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b1e9-6466c548d-tzkqd"]
Sep 08 00:54:58.154080 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:58.154044    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e040a327-067b-4512-a6b3-1aa142f8546a" path="/var/lib/kubelet/pods/e040a327-067b-4512-a6b3-1aa142f8546a/volumes"
Sep 08 00:54:59.444676 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:54:59.444640    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"]
Sep 08 00:55:04.144686 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:04.144643    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:55:14.465109 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.465003    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9" podUID="19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" containerName="main" containerID="cri-o://7c75320da1203c3f936e91205d255e937273d4e0ecc1abb29f4b4fbea2c29433" gracePeriod=45
Sep 08 00:55:14.473448 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.473406    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9" podUID="19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 00:55:14.654670 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.654637    2555 generic.go:356] "Generic (PLEG): container finished" podID="19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" containerID="7c75320da1203c3f936e91205d255e937273d4e0ecc1abb29f4b4fbea2c29433" exitCode=0
Sep 08 00:55:14.654670 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.654676    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9" event={"ID":"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e","Type":"ContainerDied","Data":"7c75320da1203c3f936e91205d255e937273d4e0ecc1abb29f4b4fbea2c29433"}
Sep 08 00:55:14.715039 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.715006    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:55:14.787890 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.787818    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-model-cache\") pod \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") "
Sep 08 00:55:14.787890 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.787855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkxwl\" (UniqueName: \"kubernetes.io/projected/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-kube-api-access-zkxwl\") pod \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") "
Sep 08 00:55:14.787890 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.787877    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-dshm\") pod \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") "
Sep 08 00:55:14.788149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.787978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tls-certs\") pod \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") "
Sep 08 00:55:14.788149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.788007    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tmp-dir\") pod \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") "
Sep 08 00:55:14.788149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.788075    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-home\") pod \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\" (UID: \"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e\") "
Sep 08 00:55:14.788149 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.788128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" (UID: "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:55:14.788370 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.788349    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:14.788422 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.788378    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" (UID: "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:55:14.788478 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.788385    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-home" (OuterVolumeSpecName: "home") pod "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" (UID: "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:55:14.790088 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.790065    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-dshm" (OuterVolumeSpecName: "dshm") pod "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" (UID: "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:55:14.790201 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.790118    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-kube-api-access-zkxwl" (OuterVolumeSpecName: "kube-api-access-zkxwl") pod "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" (UID: "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e"). InnerVolumeSpecName "kube-api-access-zkxwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:55:14.790380 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.790365    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" (UID: "19bb7388-4f43-4238-b1ea-1ce8a05ffc0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:55:14.889023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.888996    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxwl\" (UniqueName: \"kubernetes.io/projected/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-kube-api-access-zkxwl\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:14.889023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.889020    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:14.889023 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.889030    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:14.889218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.889038    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:14.889218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:14.889046    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:15.661713 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:15.661674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9" event={"ID":"19bb7388-4f43-4238-b1ea-1ce8a05ffc0e","Type":"ContainerDied","Data":"ee4916a340ecf5d97058d3068c41a2732c4e473022ee66228833a32d869f8b6e"}
Sep 08 00:55:15.662119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:15.661721    2555 scope.go:117] "RemoveContainer" containerID="7c75320da1203c3f936e91205d255e937273d4e0ecc1abb29f4b4fbea2c29433"
Sep 08 00:55:15.662119 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:15.661683    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"
Sep 08 00:55:15.682882 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:15.682857    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"]
Sep 08 00:55:15.686363 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:15.686340    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b8c4de/auth-test-model-b8c4de-kserve-699c89b86d-bmbv9"]
Sep 08 00:55:16.153438 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:16.153401    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" path="/var/lib/kubelet/pods/19bb7388-4f43-4238-b1ea-1ce8a05ffc0e/volumes"
Sep 08 00:55:18.627022 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:18.626981    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:55:18.973813 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:18.973734    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"]
Sep 08 00:55:18.973976 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:18.973952    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w" podUID="af39042e-732d-4541-9af7-ee4ef35cb506" containerName="maas-api" containerID="cri-o://d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef" gracePeriod=30
Sep 08 00:55:19.229361 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.229301    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:55:19.330062 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.330022    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af39042e-732d-4541-9af7-ee4ef35cb506-openshift-service-ca\") pod \"af39042e-732d-4541-9af7-ee4ef35cb506\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") "
Sep 08 00:55:19.330249 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.330104    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls\") pod \"af39042e-732d-4541-9af7-ee4ef35cb506\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") "
Sep 08 00:55:19.330249 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.330166    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-metrics-certs\") pod \"af39042e-732d-4541-9af7-ee4ef35cb506\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") "
Sep 08 00:55:19.330377 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.330259    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc8pf\" (UniqueName: \"kubernetes.io/projected/af39042e-732d-4541-9af7-ee4ef35cb506-kube-api-access-wc8pf\") pod \"af39042e-732d-4541-9af7-ee4ef35cb506\" (UID: \"af39042e-732d-4541-9af7-ee4ef35cb506\") "
Sep 08 00:55:19.330441 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.330420    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af39042e-732d-4541-9af7-ee4ef35cb506-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af39042e-732d-4541-9af7-ee4ef35cb506" (UID: "af39042e-732d-4541-9af7-ee4ef35cb506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:55:19.330673 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.330649    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af39042e-732d-4541-9af7-ee4ef35cb506-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:19.332516 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.332485    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af39042e-732d-4541-9af7-ee4ef35cb506" (UID: "af39042e-732d-4541-9af7-ee4ef35cb506"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:55:19.332644 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.332515    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af39042e-732d-4541-9af7-ee4ef35cb506-kube-api-access-wc8pf" (OuterVolumeSpecName: "kube-api-access-wc8pf") pod "af39042e-732d-4541-9af7-ee4ef35cb506" (UID: "af39042e-732d-4541-9af7-ee4ef35cb506"). InnerVolumeSpecName "kube-api-access-wc8pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:55:19.332644 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.332551    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af39042e-732d-4541-9af7-ee4ef35cb506" (UID: "af39042e-732d-4541-9af7-ee4ef35cb506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:55:19.431465 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.431428    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:19.431465 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.431464    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af39042e-732d-4541-9af7-ee4ef35cb506-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:19.431736 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.431478    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc8pf\" (UniqueName: \"kubernetes.io/projected/af39042e-732d-4541-9af7-ee4ef35cb506-kube-api-access-wc8pf\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:19.684906 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.684869    2555 generic.go:356] "Generic (PLEG): container finished" podID="af39042e-732d-4541-9af7-ee4ef35cb506" containerID="d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef" exitCode=0
Sep 08 00:55:19.685309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.684916    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w" event={"ID":"af39042e-732d-4541-9af7-ee4ef35cb506","Type":"ContainerDied","Data":"d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef"}
Sep 08 00:55:19.685309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.684931    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"
Sep 08 00:55:19.685309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.684982    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w" event={"ID":"af39042e-732d-4541-9af7-ee4ef35cb506","Type":"ContainerDied","Data":"86f9dafb7bf3ba5fac422bef9a3e0ad9e7740f734d3444a8b6e651dfd6ef27b0"}
Sep 08 00:55:19.685309 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.685004    2555 scope.go:117] "RemoveContainer" containerID="d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef"
Sep 08 00:55:19.694970 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.694953    2555 scope.go:117] "RemoveContainer" containerID="d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef"
Sep 08 00:55:19.695251 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:55:19.695235    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef\": container with ID starting with d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef not found: ID does not exist" containerID="d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef"
Sep 08 00:55:19.695291 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.695261    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef"} err="failed to get container status \"d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef\": rpc error: code = NotFound desc = could not find container \"d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef\": container with ID starting with d05013c9dac908d4001cfe11bbdc0a10008d6ceacc053250fbbdaf8fdbfce6ef not found: ID does not exist"
Sep 08 00:55:19.706007 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.705982    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"]
Sep 08 00:55:19.709797 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:19.709765    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e8800-56655f8ffd-qqn2w"]
Sep 08 00:55:20.153413 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:20.153379    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af39042e-732d-4541-9af7-ee4ef35cb506" path="/var/lib/kubelet/pods/af39042e-732d-4541-9af7-ee4ef35cb506/volumes"
Sep 08 00:55:32.730213 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:32.730180    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:55:46.371336 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:46.371305    2555 scope.go:117] "RemoveContainer" containerID="ebd5af503dd387ff6dfb6aa99505ce67f0109fd6b1edaf3ef2f493157e444240"
Sep 08 00:55:50.126418 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:50.126381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:55:54.116558 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.116517    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"]
Sep 08 00:55:54.117054 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.116792    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5" podUID="61dc3c05-6f53-43ad-bb96-a0e09dca0b23" containerName="maas-api" containerID="cri-o://805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d" gracePeriod=30
Sep 08 00:55:54.367024 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.366969    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:55:54.451347 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.451306    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-metrics-certs\") pod \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") "
Sep 08 00:55:54.451520 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.451381    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-maas-api-tls\") pod \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") "
Sep 08 00:55:54.451520 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.451431    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jltcq\" (UniqueName: \"kubernetes.io/projected/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-kube-api-access-jltcq\") pod \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") "
Sep 08 00:55:54.451644 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.451529    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-openshift-service-ca\") pod \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\" (UID: \"61dc3c05-6f53-43ad-bb96-a0e09dca0b23\") "
Sep 08 00:55:54.451999 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.451960    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61dc3c05-6f53-43ad-bb96-a0e09dca0b23" (UID: "61dc3c05-6f53-43ad-bb96-a0e09dca0b23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:55:54.453359 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.453332    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "61dc3c05-6f53-43ad-bb96-a0e09dca0b23" (UID: "61dc3c05-6f53-43ad-bb96-a0e09dca0b23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:55:54.453482 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.453354    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61dc3c05-6f53-43ad-bb96-a0e09dca0b23" (UID: "61dc3c05-6f53-43ad-bb96-a0e09dca0b23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:55:54.453537 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.453518    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-kube-api-access-jltcq" (OuterVolumeSpecName: "kube-api-access-jltcq") pod "61dc3c05-6f53-43ad-bb96-a0e09dca0b23" (UID: "61dc3c05-6f53-43ad-bb96-a0e09dca0b23"). InnerVolumeSpecName "kube-api-access-jltcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:55:54.553190 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.553153    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:54.553190 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.553185    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:54.553190 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.553199    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:54.553417 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.553206    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jltcq\" (UniqueName: \"kubernetes.io/projected/61dc3c05-6f53-43ad-bb96-a0e09dca0b23-kube-api-access-jltcq\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:54.884317 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.884281    2555 generic.go:356] "Generic (PLEG): container finished" podID="61dc3c05-6f53-43ad-bb96-a0e09dca0b23" containerID="805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d" exitCode=0
Sep 08 00:55:54.884484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.884326    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"
Sep 08 00:55:54.884484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.884334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5" event={"ID":"61dc3c05-6f53-43ad-bb96-a0e09dca0b23","Type":"ContainerDied","Data":"805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d"}
Sep 08 00:55:54.884484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.884375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5" event={"ID":"61dc3c05-6f53-43ad-bb96-a0e09dca0b23","Type":"ContainerDied","Data":"634eabbb93313d75948837253b1196acee60b5bcd5880b7ca6b6c8fb9c824cdf"}
Sep 08 00:55:54.884484 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.884396    2555 scope.go:117] "RemoveContainer" containerID="805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d"
Sep 08 00:55:54.893870 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.893850    2555 scope.go:117] "RemoveContainer" containerID="805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d"
Sep 08 00:55:54.894134 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:55:54.894111    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d\": container with ID starting with 805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d not found: ID does not exist" containerID="805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d"
Sep 08 00:55:54.894189 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.894142    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d"} err="failed to get container status \"805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d\": rpc error: code = NotFound desc = could not find container \"805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d\": container with ID starting with 805b14088c07ef3753c1f62d9d09bc9f46a1c6b6d149268576010ea0518a7a0d not found: ID does not exist"
Sep 08 00:55:54.905124 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.905101    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"]
Sep 08 00:55:54.909051 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:54.909030    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f04ef7-58b569489f-pn5v5"]
Sep 08 00:55:55.261821 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.261744    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"]
Sep 08 00:55:55.262198 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.261939    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7" podUID="83c44880-38cd-4b55-a75e-f4d52776a770" containerName="maas-api" containerID="cri-o://f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e" gracePeriod=30
Sep 08 00:55:55.518898 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.518832    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:55:55.661635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.661583    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p7ls\" (UniqueName: \"kubernetes.io/projected/83c44880-38cd-4b55-a75e-f4d52776a770-kube-api-access-5p7ls\") pod \"83c44880-38cd-4b55-a75e-f4d52776a770\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") "
Sep 08 00:55:55.661823 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.661668    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-metrics-certs\") pod \"83c44880-38cd-4b55-a75e-f4d52776a770\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") "
Sep 08 00:55:55.661823 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.661690    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-maas-api-tls\") pod \"83c44880-38cd-4b55-a75e-f4d52776a770\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") "
Sep 08 00:55:55.661823 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.661755    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44880-38cd-4b55-a75e-f4d52776a770-openshift-service-ca\") pod \"83c44880-38cd-4b55-a75e-f4d52776a770\" (UID: \"83c44880-38cd-4b55-a75e-f4d52776a770\") "
Sep 08 00:55:55.662210 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.662175    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c44880-38cd-4b55-a75e-f4d52776a770-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83c44880-38cd-4b55-a75e-f4d52776a770" (UID: "83c44880-38cd-4b55-a75e-f4d52776a770"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:55:55.663750 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.663720    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "83c44880-38cd-4b55-a75e-f4d52776a770" (UID: "83c44880-38cd-4b55-a75e-f4d52776a770"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:55:55.663846 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.663753    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83c44880-38cd-4b55-a75e-f4d52776a770" (UID: "83c44880-38cd-4b55-a75e-f4d52776a770"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:55:55.663846 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.663785    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c44880-38cd-4b55-a75e-f4d52776a770-kube-api-access-5p7ls" (OuterVolumeSpecName: "kube-api-access-5p7ls") pod "83c44880-38cd-4b55-a75e-f4d52776a770" (UID: "83c44880-38cd-4b55-a75e-f4d52776a770"). InnerVolumeSpecName "kube-api-access-5p7ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:55:55.763032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.762996    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c44880-38cd-4b55-a75e-f4d52776a770-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:55.763032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.763025    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7ls\" (UniqueName: \"kubernetes.io/projected/83c44880-38cd-4b55-a75e-f4d52776a770-kube-api-access-5p7ls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:55.763032 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.763035    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:55.763264 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.763048    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c44880-38cd-4b55-a75e-f4d52776a770-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:55:55.891862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.891828    2555 generic.go:356] "Generic (PLEG): container finished" podID="83c44880-38cd-4b55-a75e-f4d52776a770" containerID="f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e" exitCode=0
Sep 08 00:55:55.892040 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.891877    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"
Sep 08 00:55:55.892040 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.891896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7" event={"ID":"83c44880-38cd-4b55-a75e-f4d52776a770","Type":"ContainerDied","Data":"f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e"}
Sep 08 00:55:55.892040 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.891948    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7" event={"ID":"83c44880-38cd-4b55-a75e-f4d52776a770","Type":"ContainerDied","Data":"e69eaccae64a54776336b530617d0f2131bb8fb1dc79566b083a90ea56dadcfe"}
Sep 08 00:55:55.892040 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.891978    2555 scope.go:117] "RemoveContainer" containerID="f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e"
Sep 08 00:55:55.901635 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.901616    2555 scope.go:117] "RemoveContainer" containerID="f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e"
Sep 08 00:55:55.901912 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:55:55.901892    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e\": container with ID starting with f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e not found: ID does not exist" containerID="f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e"
Sep 08 00:55:55.901965 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.901921    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e"} err="failed to get container status \"f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e\": rpc error: code = NotFound desc = could not find container \"f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e\": container with ID starting with f65f83fcff07114f8fe171e51c6a87156cf5d2416113d7a4c9224361dc92b48e not found: ID does not exist"
Sep 08 00:55:55.931328 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.931294    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"]
Sep 08 00:55:55.937218 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:55.937196    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e810-578fdf9677-2gvm7"]
Sep 08 00:55:56.153691 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:56.153610    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61dc3c05-6f53-43ad-bb96-a0e09dca0b23" path="/var/lib/kubelet/pods/61dc3c05-6f53-43ad-bb96-a0e09dca0b23/volumes"
Sep 08 00:55:56.154063 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:55:56.154048    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c44880-38cd-4b55-a75e-f4d52776a770" path="/var/lib/kubelet/pods/83c44880-38cd-4b55-a75e-f4d52776a770/volumes"
Sep 08 00:56:04.539397 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.539356    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-6dnk6"]
Sep 08 00:56:04.540013 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.539558    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6" podUID="e483cda4-02a1-4cdd-a652-0561e1ee1175" containerName="manager" containerID="cri-o://76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b" gracePeriod=30
Sep 08 00:56:04.782637 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.782574    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:56:04.944398 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.944362    2555 generic.go:356] "Generic (PLEG): container finished" podID="e483cda4-02a1-4cdd-a652-0561e1ee1175" containerID="76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b" exitCode=0
Sep 08 00:56:04.944571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.944412    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6"
Sep 08 00:56:04.944571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.944447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6" event={"ID":"e483cda4-02a1-4cdd-a652-0561e1ee1175","Type":"ContainerDied","Data":"76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b"}
Sep 08 00:56:04.944571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.944504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-6dnk6" event={"ID":"e483cda4-02a1-4cdd-a652-0561e1ee1175","Type":"ContainerDied","Data":"1d1a62b33dde05e5388342f7f7fec21064b3f21fd3569e4a52fa5753c2700334"}
Sep 08 00:56:04.944571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.944526    2555 scope.go:117] "RemoveContainer" containerID="76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b"
Sep 08 00:56:04.945683 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.945666    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-webhook-cert\") pod \"e483cda4-02a1-4cdd-a652-0561e1ee1175\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") "
Sep 08 00:56:04.945777 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.945731    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtvl4\" (UniqueName: \"kubernetes.io/projected/e483cda4-02a1-4cdd-a652-0561e1ee1175-kube-api-access-vtvl4\") pod \"e483cda4-02a1-4cdd-a652-0561e1ee1175\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") "
Sep 08 00:56:04.945840 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.945778    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-metrics-certs\") pod \"e483cda4-02a1-4cdd-a652-0561e1ee1175\" (UID: \"e483cda4-02a1-4cdd-a652-0561e1ee1175\") "
Sep 08 00:56:04.947862 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.947837    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e483cda4-02a1-4cdd-a652-0561e1ee1175" (UID: "e483cda4-02a1-4cdd-a652-0561e1ee1175"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:56:04.947967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.947834    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e483cda4-02a1-4cdd-a652-0561e1ee1175-kube-api-access-vtvl4" (OuterVolumeSpecName: "kube-api-access-vtvl4") pod "e483cda4-02a1-4cdd-a652-0561e1ee1175" (UID: "e483cda4-02a1-4cdd-a652-0561e1ee1175"). InnerVolumeSpecName "kube-api-access-vtvl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:56:04.947967 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.947937    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e483cda4-02a1-4cdd-a652-0561e1ee1175" (UID: "e483cda4-02a1-4cdd-a652-0561e1ee1175"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:56:04.965781 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.965761    2555 scope.go:117] "RemoveContainer" containerID="76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b"
Sep 08 00:56:04.966068 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:56:04.966047    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b\": container with ID starting with 76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b not found: ID does not exist" containerID="76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b"
Sep 08 00:56:04.966134 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:04.966075    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b"} err="failed to get container status \"76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b\": rpc error: code = NotFound desc = could not find container \"76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b\": container with ID starting with 76e583227e35ae495aa3ed5d27855cae99eb594049defce1566425799f93626b not found: ID does not exist"
Sep 08 00:56:05.047006 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:05.046972    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvl4\" (UniqueName: \"kubernetes.io/projected/e483cda4-02a1-4cdd-a652-0561e1ee1175-kube-api-access-vtvl4\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:56:05.047006 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:05.047004    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:56:05.047006 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:05.047013    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e483cda4-02a1-4cdd-a652-0561e1ee1175-webhook-cert\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:56:05.266389 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:05.266359    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-6dnk6"]
Sep 08 00:56:05.269650 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:05.269627    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-6dnk6"]
Sep 08 00:56:05.832571 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:05.832531    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:56:06.154688 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:06.154607    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e483cda4-02a1-4cdd-a652-0561e1ee1175" path="/var/lib/kubelet/pods/e483cda4-02a1-4cdd-a652-0561e1ee1175/volumes"
Sep 08 00:56:17.634188 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:56:17.634154    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:57:06.015084 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015049    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-q2t6w"]
Sep 08 00:57:06.015540 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015484    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9279ece-eb66-4459-8396-85c1b519a07e" containerName="maas-api"
Sep 08 00:57:06.015540 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015501    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9279ece-eb66-4459-8396-85c1b519a07e" containerName="maas-api"
Sep 08 00:57:06.015540 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015512    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af39042e-732d-4541-9af7-ee4ef35cb506" containerName="maas-api"
Sep 08 00:57:06.015540 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015521    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af39042e-732d-4541-9af7-ee4ef35cb506" containerName="maas-api"
Sep 08 00:57:06.015540 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015538    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61dc3c05-6f53-43ad-bb96-a0e09dca0b23" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015547    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="61dc3c05-6f53-43ad-bb96-a0e09dca0b23" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015558    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e040a327-067b-4512-a6b3-1aa142f8546a" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015566    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e040a327-067b-4512-a6b3-1aa142f8546a" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015585    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e483cda4-02a1-4cdd-a652-0561e1ee1175" containerName="manager"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015608    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e483cda4-02a1-4cdd-a652-0561e1ee1175" containerName="manager"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015621    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c44880-38cd-4b55-a75e-f4d52776a770" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015629    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c44880-38cd-4b55-a75e-f4d52776a770" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015640    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" containerName="main"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015647    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" containerName="main"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015723    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9279ece-eb66-4459-8396-85c1b519a07e" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015731    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="61dc3c05-6f53-43ad-bb96-a0e09dca0b23" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015738    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="af39042e-732d-4541-9af7-ee4ef35cb506" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015747    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bb7388-4f43-4238-b1ea-1ce8a05ffc0e" containerName="main"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015755    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e040a327-067b-4512-a6b3-1aa142f8546a" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015762    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c44880-38cd-4b55-a75e-f4d52776a770" containerName="maas-api"
Sep 08 00:57:06.015769 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.015771    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e483cda4-02a1-4cdd-a652-0561e1ee1175" containerName="manager"
Sep 08 00:57:06.020373 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.020356    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.022977 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.022952    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 00:57:06.024173 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.024155    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 00:57:06.024265 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.024216    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddxg8\""
Sep 08 00:57:06.024265 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.024215    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 00:57:06.028832 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.028812    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-q2t6w"]
Sep 08 00:57:06.187511 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.187472    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.187511 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.187514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.187511 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.187531    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zkd\" (UniqueName: \"kubernetes.io/projected/463e495e-2e65-4ed4-aa76-16ab77ce7889-kube-api-access-q4zkd\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.288462 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.288380    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.288462 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.288412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.288462 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.288452    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zkd\" (UniqueName: \"kubernetes.io/projected/463e495e-2e65-4ed4-aa76-16ab77ce7889-kube-api-access-q4zkd\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.290894 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.290869    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.291015 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.290908    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.296224 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.296204    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zkd\" (UniqueName: \"kubernetes.io/projected/463e495e-2e65-4ed4-aa76-16ab77ce7889-kube-api-access-q4zkd\") pod \"maas-controller-5dbbcf5b8c-q2t6w\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") " pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.332499 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.332473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:06.600904 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.600880    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-q2t6w"]
Sep 08 00:57:06.602740 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:57:06.602710    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463e495e_2e65_4ed4_aa76_16ab77ce7889.slice/crio-6dda8e1a40982b5d2b25f527450c37c28da6194e12cb62f18535722d7e11fce6 WatchSource:0}: Error finding container 6dda8e1a40982b5d2b25f527450c37c28da6194e12cb62f18535722d7e11fce6: Status 404 returned error can't find the container with id 6dda8e1a40982b5d2b25f527450c37c28da6194e12cb62f18535722d7e11fce6
Sep 08 00:57:06.603918 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:06.603900    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:57:07.274421 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:07.274384    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w" event={"ID":"463e495e-2e65-4ed4-aa76-16ab77ce7889","Type":"ContainerStarted","Data":"002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909"}
Sep 08 00:57:07.274858 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:07.274447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w" event={"ID":"463e495e-2e65-4ed4-aa76-16ab77ce7889","Type":"ContainerStarted","Data":"6dda8e1a40982b5d2b25f527450c37c28da6194e12cb62f18535722d7e11fce6"}
Sep 08 00:57:07.274858 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:07.274479    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:07.296179 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:07.296130    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w" podStartSLOduration=1.8149578 podStartE2EDuration="2.296114492s" totalImagesPullingTime="481.156692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:57:07.291480226 +0000 UTC m=+1461.634783246" watchObservedRunningTime="2026-09-08 00:57:07.296114492 +0000 UTC m=+1461.639417539"
Sep 08 00:57:18.282050 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:18.282018    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:29.629943 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.629911    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-q2t6w"]
Sep 08 00:57:29.630389 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.630127    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w" podUID="463e495e-2e65-4ed4-aa76-16ab77ce7889" containerName="manager" containerID="cri-o://002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909" gracePeriod=30
Sep 08 00:57:29.877306 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.877278    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:29.979795 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.979712    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4zkd\" (UniqueName: \"kubernetes.io/projected/463e495e-2e65-4ed4-aa76-16ab77ce7889-kube-api-access-q4zkd\") pod \"463e495e-2e65-4ed4-aa76-16ab77ce7889\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") "
Sep 08 00:57:29.979795 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.979781    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-metrics-certs\") pod \"463e495e-2e65-4ed4-aa76-16ab77ce7889\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") "
Sep 08 00:57:29.980005 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.979877    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-webhook-cert\") pod \"463e495e-2e65-4ed4-aa76-16ab77ce7889\" (UID: \"463e495e-2e65-4ed4-aa76-16ab77ce7889\") "
Sep 08 00:57:29.981751 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.981721    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "463e495e-2e65-4ed4-aa76-16ab77ce7889" (UID: "463e495e-2e65-4ed4-aa76-16ab77ce7889"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:57:29.981884 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.981815    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463e495e-2e65-4ed4-aa76-16ab77ce7889-kube-api-access-q4zkd" (OuterVolumeSpecName: "kube-api-access-q4zkd") pod "463e495e-2e65-4ed4-aa76-16ab77ce7889" (UID: "463e495e-2e65-4ed4-aa76-16ab77ce7889"). InnerVolumeSpecName "kube-api-access-q4zkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:57:29.981884 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:29.981835    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "463e495e-2e65-4ed4-aa76-16ab77ce7889" (UID: "463e495e-2e65-4ed4-aa76-16ab77ce7889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:57:30.081422 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.081386    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-webhook-cert\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:57:30.081422 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.081417    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4zkd\" (UniqueName: \"kubernetes.io/projected/463e495e-2e65-4ed4-aa76-16ab77ce7889-kube-api-access-q4zkd\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:57:30.081619 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.081430    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/463e495e-2e65-4ed4-aa76-16ab77ce7889-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 00:57:30.393205 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.393171    2555 generic.go:356] "Generic (PLEG): container finished" podID="463e495e-2e65-4ed4-aa76-16ab77ce7889" containerID="002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909" exitCode=0
Sep 08 00:57:30.393339 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.393221    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w"
Sep 08 00:57:30.393339 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.393216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w" event={"ID":"463e495e-2e65-4ed4-aa76-16ab77ce7889","Type":"ContainerDied","Data":"002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909"}
Sep 08 00:57:30.393430 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.393345    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-q2t6w" event={"ID":"463e495e-2e65-4ed4-aa76-16ab77ce7889","Type":"ContainerDied","Data":"6dda8e1a40982b5d2b25f527450c37c28da6194e12cb62f18535722d7e11fce6"}
Sep 08 00:57:30.393430 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.393375    2555 scope.go:117] "RemoveContainer" containerID="002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909"
Sep 08 00:57:30.402062 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.402048    2555 scope.go:117] "RemoveContainer" containerID="002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909"
Sep 08 00:57:30.402326 ip-10-0-142-98 kubenswrapper[2555]: E0908 00:57:30.402305    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909\": container with ID starting with 002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909 not found: ID does not exist" containerID="002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909"
Sep 08 00:57:30.402410 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.402335    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909"} err="failed to get container status \"002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909\": rpc error: code = NotFound desc = could not find container \"002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909\": container with ID starting with 002a80543e1390a106566751917635383f1dd45e0782ef9dde9896bdd66af909 not found: ID does not exist"
Sep 08 00:57:30.409542 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.409516    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-q2t6w"]
Sep 08 00:57:30.413474 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:30.413454    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-q2t6w"]
Sep 08 00:57:31.128037 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.128001    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-b9glp"]
Sep 08 00:57:31.128423 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.128406    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463e495e-2e65-4ed4-aa76-16ab77ce7889" containerName="manager"
Sep 08 00:57:31.128473 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.128425    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="463e495e-2e65-4ed4-aa76-16ab77ce7889" containerName="manager"
Sep 08 00:57:31.128515 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.128506    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="463e495e-2e65-4ed4-aa76-16ab77ce7889" containerName="manager"
Sep 08 00:57:31.132834 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.132813    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.136437 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.136418    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 00:57:31.136437 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.136441    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 00:57:31.136437 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.136445    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddxg8\""
Sep 08 00:57:31.136677 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.136510    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 00:57:31.141159 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.141137    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-b9glp"]
Sep 08 00:57:31.190510 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.190486    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/158bb49c-6f6e-4e7b-99b4-15e93f86478f-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.190641 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.190516    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ml5s\" (UniqueName: \"kubernetes.io/projected/158bb49c-6f6e-4e7b-99b4-15e93f86478f-kube-api-access-5ml5s\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.190641 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.190576    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158bb49c-6f6e-4e7b-99b4-15e93f86478f-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.291548 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.291521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/158bb49c-6f6e-4e7b-99b4-15e93f86478f-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.291675 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.291551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ml5s\" (UniqueName: \"kubernetes.io/projected/158bb49c-6f6e-4e7b-99b4-15e93f86478f-kube-api-access-5ml5s\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.291675 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.291579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158bb49c-6f6e-4e7b-99b4-15e93f86478f-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.293949 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.293921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/158bb49c-6f6e-4e7b-99b4-15e93f86478f-webhook-cert\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.294050 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.293963    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158bb49c-6f6e-4e7b-99b4-15e93f86478f-metrics-certs\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.299265 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.299246    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ml5s\" (UniqueName: \"kubernetes.io/projected/158bb49c-6f6e-4e7b-99b4-15e93f86478f-kube-api-access-5ml5s\") pod \"maas-controller-5dbbcf5b8c-b9glp\" (UID: \"158bb49c-6f6e-4e7b-99b4-15e93f86478f\") " pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.444553 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.444490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:31.715541 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:31.715515    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dbbcf5b8c-b9glp"]
Sep 08 00:57:31.717547 ip-10-0-142-98 kubenswrapper[2555]: W0908 00:57:31.717509    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158bb49c_6f6e_4e7b_99b4_15e93f86478f.slice/crio-a673109924907da9badd52014d00291377fd3f85d6fd7880a1d0fc6ea18a214e WatchSource:0}: Error finding container a673109924907da9badd52014d00291377fd3f85d6fd7880a1d0fc6ea18a214e: Status 404 returned error can't find the container with id a673109924907da9badd52014d00291377fd3f85d6fd7880a1d0fc6ea18a214e
Sep 08 00:57:32.160871 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:32.160835    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="463e495e-2e65-4ed4-aa76-16ab77ce7889" path="/var/lib/kubelet/pods/463e495e-2e65-4ed4-aa76-16ab77ce7889/volumes"
Sep 08 00:57:32.406709 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:32.406582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp" event={"ID":"158bb49c-6f6e-4e7b-99b4-15e93f86478f","Type":"ContainerStarted","Data":"f505e4c729e3453fa769d564eef5efed5198fa65491e514aef075f52bdf600e7"}
Sep 08 00:57:32.406709 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:32.406676    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp" event={"ID":"158bb49c-6f6e-4e7b-99b4-15e93f86478f","Type":"ContainerStarted","Data":"a673109924907da9badd52014d00291377fd3f85d6fd7880a1d0fc6ea18a214e"}
Sep 08 00:57:32.406709 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:32.406702    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:32.444543 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:32.444495    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp" podStartSLOduration=1.035695689 podStartE2EDuration="1.444479172s" totalImagesPullingTime="408.783483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:57:32.44206471 +0000 UTC m=+1486.785367729" watchObservedRunningTime="2026-09-08 00:57:32.444479172 +0000 UTC m=+1486.787782223"
Sep 08 00:57:34.028253 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:34.028216    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:57:43.415077 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:43.415049    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dbbcf5b8c-b9glp"
Sep 08 00:57:46.223630 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:46.223580    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:57:46.227340 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:46.227318    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 00:57:58.438535 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:57:58.438497    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:58:09.036182 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:58:09.036100    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:58:22.725628 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:58:22.725572    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 00:58:51.132199 ip-10-0-142-98 kubenswrapper[2555]: I0908 00:58:51.132163    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:00:00.138761 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:00.138725    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p"]
Sep 08 01:00:00.142563 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:00.142539    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p"
Sep 08 01:00:00.145322 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:00.145298    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\""
Sep 08 01:00:00.154541 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:00.154518    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p"
Sep 08 01:00:00.155848 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:00.155827    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p"]
Sep 08 01:00:00.423015 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:00.422990    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p"]
Sep 08 01:00:00.425736 ip-10-0-142-98 kubenswrapper[2555]: W0908 01:00:00.425707    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399337fd_4ea3_4ced_bec5_270b15fd1747.slice/crio-5cd366768075f60f71e7b3aab74a87ad43e86531912a03f2c70248bd172440dc WatchSource:0}: Error finding container 5cd366768075f60f71e7b3aab74a87ad43e86531912a03f2c70248bd172440dc: Status 404 returned error can't find the container with id 5cd366768075f60f71e7b3aab74a87ad43e86531912a03f2c70248bd172440dc
Sep 08 01:00:01.189195 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:01.189156    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p" event={"ID":"399337fd-4ea3-4ced-bec5-270b15fd1747","Type":"ContainerStarted","Data":"784d8f19b0399264e9d0481cff620663140c641f8f27b295c3ed305709ec645a"}
Sep 08 01:00:01.189575 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:01.189212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p" event={"ID":"399337fd-4ea3-4ced-bec5-270b15fd1747","Type":"ContainerStarted","Data":"5cd366768075f60f71e7b3aab74a87ad43e86531912a03f2c70248bd172440dc"}
Sep 08 01:00:01.206019 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:01.205964    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p" podStartSLOduration=1.205948636 podStartE2EDuration="1.205948636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:00:01.204710056 +0000 UTC m=+1635.548013096" watchObservedRunningTime="2026-09-08 01:00:01.205948636 +0000 UTC m=+1635.549251654"
Sep 08 01:00:02.195074 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:02.195040    2555 generic.go:356] "Generic (PLEG): container finished" podID="399337fd-4ea3-4ced-bec5-270b15fd1747" containerID="784d8f19b0399264e9d0481cff620663140c641f8f27b295c3ed305709ec645a" exitCode=0
Sep 08 01:00:02.195462 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:02.195088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p" event={"ID":"399337fd-4ea3-4ced-bec5-270b15fd1747","Type":"ContainerDied","Data":"784d8f19b0399264e9d0481cff620663140c641f8f27b295c3ed305709ec645a"}
Sep 08 01:00:03.329679 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:03.329654    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p"
Sep 08 01:00:04.207314 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:04.207285    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p"
Sep 08 01:00:04.207468 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:04.207288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813820-nzz4p" event={"ID":"399337fd-4ea3-4ced-bec5-270b15fd1747","Type":"ContainerDied","Data":"5cd366768075f60f71e7b3aab74a87ad43e86531912a03f2c70248bd172440dc"}
Sep 08 01:00:04.207468 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:04.207408    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cd366768075f60f71e7b3aab74a87ad43e86531912a03f2c70248bd172440dc"
Sep 08 01:00:24.429570 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:24.429525    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:00:37.529656 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:37.529613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:00:46.418541 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.418506    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"]
Sep 08 01:00:46.419075 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.419053    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399337fd-4ea3-4ced-bec5-270b15fd1747" containerName="cleanup"
Sep 08 01:00:46.419075 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.419075    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="399337fd-4ea3-4ced-bec5-270b15fd1747" containerName="cleanup"
Sep 08 01:00:46.419196 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.419137    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="399337fd-4ea3-4ced-bec5-270b15fd1747" containerName="cleanup"
Sep 08 01:00:46.427430 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.427044    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.431026 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.431003    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8fc557f8-kserve-self-signed-certs\""
Sep 08 01:00:46.432383 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.432359    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"]
Sep 08 01:00:46.528256 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.528218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmbm\" (UniqueName: \"kubernetes.io/projected/1b607e6c-6c33-4acf-b312-1188fdaa4458-kube-api-access-4nmbm\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.528426 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.528284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-dshm\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.528426 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.528339    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-tmp-dir\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.528426 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.528392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-home\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.528554 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.528454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-model-cache\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.528554 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.528481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b607e6c-6c33-4acf-b312-1188fdaa4458-tls-certs\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629320 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-dshm\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629519 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629354    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-tmp-dir\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629519 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629377    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-home\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629519 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-model-cache\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629519 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b607e6c-6c33-4acf-b312-1188fdaa4458-tls-certs\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629519 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629461    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmbm\" (UniqueName: \"kubernetes.io/projected/1b607e6c-6c33-4acf-b312-1188fdaa4458-kube-api-access-4nmbm\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629892 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-home\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629892 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629878    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-model-cache\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.629892 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.629892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-tmp-dir\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.631713 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.631687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-dshm\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.632063 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.632040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b607e6c-6c33-4acf-b312-1188fdaa4458-tls-certs\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.637253 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.637232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmbm\" (UniqueName: \"kubernetes.io/projected/1b607e6c-6c33-4acf-b312-1188fdaa4458-kube-api-access-4nmbm\") pod \"e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") " pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:46.738274 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:46.738203    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:47.013195 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:47.013166    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"]
Sep 08 01:00:47.014756 ip-10-0-142-98 kubenswrapper[2555]: W0908 01:00:47.014729    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b607e6c_6c33_4acf_b312_1188fdaa4458.slice/crio-d973f4c87f59b6d5aac1dd28d01052d3e9feaf069d545ad54a679c27b9923454 WatchSource:0}: Error finding container d973f4c87f59b6d5aac1dd28d01052d3e9feaf069d545ad54a679c27b9923454: Status 404 returned error can't find the container with id d973f4c87f59b6d5aac1dd28d01052d3e9feaf069d545ad54a679c27b9923454
Sep 08 01:00:47.441713 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:47.441665    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8" event={"ID":"1b607e6c-6c33-4acf-b312-1188fdaa4458","Type":"ContainerStarted","Data":"5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103"}
Sep 08 01:00:47.441713 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:47.441715    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8" event={"ID":"1b607e6c-6c33-4acf-b312-1188fdaa4458","Type":"ContainerStarted","Data":"d973f4c87f59b6d5aac1dd28d01052d3e9feaf069d545ad54a679c27b9923454"}
Sep 08 01:00:47.459104 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:47.459052    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8" podStartSLOduration=1.459037564 podStartE2EDuration="1.459037564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:00:47.457664217 +0000 UTC m=+1681.800967273" watchObservedRunningTime="2026-09-08 01:00:47.459037564 +0000 UTC m=+1681.802340580"
Sep 08 01:00:47.738762 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:47.738686    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:50.934337 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:50.934300    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:00:56.738823 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:56.738794    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:56.751641 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:56.751605    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:00:57.506808 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:00:57.506781    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:01:05.134089 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:05.134049    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:01:23.248824 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:23.248740    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"]
Sep 08 01:01:29.738957 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:29.738923    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:01:38.268508 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.268442    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8" podUID="1b607e6c-6c33-4acf-b312-1188fdaa4458" containerName="main" containerID="cri-o://5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103" gracePeriod=45
Sep 08 01:01:38.512262 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.512233    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:01:38.594146 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594115    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b607e6c-6c33-4acf-b312-1188fdaa4458-tls-certs\") pod \"1b607e6c-6c33-4acf-b312-1188fdaa4458\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") "
Sep 08 01:01:38.594307 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594187    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nmbm\" (UniqueName: \"kubernetes.io/projected/1b607e6c-6c33-4acf-b312-1188fdaa4458-kube-api-access-4nmbm\") pod \"1b607e6c-6c33-4acf-b312-1188fdaa4458\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") "
Sep 08 01:01:38.594307 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594218    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-home\") pod \"1b607e6c-6c33-4acf-b312-1188fdaa4458\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") "
Sep 08 01:01:38.594307 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594249    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-dshm\") pod \"1b607e6c-6c33-4acf-b312-1188fdaa4458\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") "
Sep 08 01:01:38.594307 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594265    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-tmp-dir\") pod \"1b607e6c-6c33-4acf-b312-1188fdaa4458\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") "
Sep 08 01:01:38.594530 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594341    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-model-cache\") pod \"1b607e6c-6c33-4acf-b312-1188fdaa4458\" (UID: \"1b607e6c-6c33-4acf-b312-1188fdaa4458\") "
Sep 08 01:01:38.594619 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594543    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-home" (OuterVolumeSpecName: "home") pod "1b607e6c-6c33-4acf-b312-1188fdaa4458" (UID: "1b607e6c-6c33-4acf-b312-1188fdaa4458"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 01:01:38.594619 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594548    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b607e6c-6c33-4acf-b312-1188fdaa4458" (UID: "1b607e6c-6c33-4acf-b312-1188fdaa4458"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 01:01:38.594704 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594622    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b607e6c-6c33-4acf-b312-1188fdaa4458" (UID: "1b607e6c-6c33-4acf-b312-1188fdaa4458"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 01:01:38.594704 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594689    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-tmp-dir\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:38.594779 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594708    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-model-cache\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:38.594779 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.594721    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-home\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:38.596359 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.596336    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b607e6c-6c33-4acf-b312-1188fdaa4458-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b607e6c-6c33-4acf-b312-1188fdaa4458" (UID: "1b607e6c-6c33-4acf-b312-1188fdaa4458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 01:01:38.596441 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.596378    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-dshm" (OuterVolumeSpecName: "dshm") pod "1b607e6c-6c33-4acf-b312-1188fdaa4458" (UID: "1b607e6c-6c33-4acf-b312-1188fdaa4458"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 01:01:38.596482 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.596436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b607e6c-6c33-4acf-b312-1188fdaa4458-kube-api-access-4nmbm" (OuterVolumeSpecName: "kube-api-access-4nmbm") pod "1b607e6c-6c33-4acf-b312-1188fdaa4458" (UID: "1b607e6c-6c33-4acf-b312-1188fdaa4458"). InnerVolumeSpecName "kube-api-access-4nmbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 01:01:38.695812 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.695776    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmbm\" (UniqueName: \"kubernetes.io/projected/1b607e6c-6c33-4acf-b312-1188fdaa4458-kube-api-access-4nmbm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:38.695812 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.695806    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b607e6c-6c33-4acf-b312-1188fdaa4458-dshm\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:38.695812 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.695818    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b607e6c-6c33-4acf-b312-1188fdaa4458-tls-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:38.709511 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.709481    2555 generic.go:356] "Generic (PLEG): container finished" podID="1b607e6c-6c33-4acf-b312-1188fdaa4458" containerID="5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103" exitCode=0
Sep 08 01:01:38.709511 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.709514    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8" event={"ID":"1b607e6c-6c33-4acf-b312-1188fdaa4458","Type":"ContainerDied","Data":"5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103"}
Sep 08 01:01:38.709753 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.709533    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"
Sep 08 01:01:38.709753 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.709551    2555 scope.go:117] "RemoveContainer" containerID="5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103"
Sep 08 01:01:38.709753 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.709541    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8" event={"ID":"1b607e6c-6c33-4acf-b312-1188fdaa4458","Type":"ContainerDied","Data":"d973f4c87f59b6d5aac1dd28d01052d3e9feaf069d545ad54a679c27b9923454"}
Sep 08 01:01:38.718992 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.718974    2555 scope.go:117] "RemoveContainer" containerID="5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103"
Sep 08 01:01:38.719223 ip-10-0-142-98 kubenswrapper[2555]: E0908 01:01:38.719205    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103\": container with ID starting with 5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103 not found: ID does not exist" containerID="5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103"
Sep 08 01:01:38.719263 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.719232    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103"} err="failed to get container status \"5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103\": rpc error: code = NotFound desc = could not find container \"5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103\": container with ID starting with 5b3864ea819c19c1683c92b48652e93223a8128e0d657a67bd5283bcd4c85103 not found: ID does not exist"
Sep 08 01:01:38.729934 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.729904    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"]
Sep 08 01:01:38.737721 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:38.737700    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8"]
Sep 08 01:01:39.496391 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:39.496350    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-8fc557f8-kserve-58b67679f6-9hgz8" podUID="1b607e6c-6c33-4acf-b312-1188fdaa4458" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Sep 08 01:01:40.153179 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:40.153147    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b607e6c-6c33-4acf-b312-1188fdaa4458" path="/var/lib/kubelet/pods/1b607e6c-6c33-4acf-b312-1188fdaa4458/volumes"
Sep 08 01:01:42.335810 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:42.335774    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:01:51.828026 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.827987    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"]
Sep 08 01:01:51.828632 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.828612    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b607e6c-6c33-4acf-b312-1188fdaa4458" containerName="main"
Sep 08 01:01:51.828632 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.828633    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b607e6c-6c33-4acf-b312-1188fdaa4458" containerName="main"
Sep 08 01:01:51.828758 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.828720    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b607e6c-6c33-4acf-b312-1188fdaa4458" containerName="main"
Sep 08 01:01:51.831631 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.831610    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:51.835148 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.835125    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 01:01:51.835148 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.835144    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gnd2h\""
Sep 08 01:01:51.835349 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.835207    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 01:01:51.835349 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.835134    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 01:01:51.835349 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.835131    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 01:01:51.842999 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.842975    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"]
Sep 08 01:01:51.911434 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.911406    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfhq\" (UniqueName: \"kubernetes.io/projected/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-kube-api-access-czfhq\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:51.911617 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.911464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-metrics-certs\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:51.911617 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.911574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-openshift-service-ca\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:51.911741 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:51.911639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-maas-api-tls\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.012395 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.012354    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-openshift-service-ca\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.012627 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.012401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-maas-api-tls\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.012627 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.012465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfhq\" (UniqueName: \"kubernetes.io/projected/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-kube-api-access-czfhq\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.012627 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.012520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-metrics-certs\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.013077 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.013056    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-openshift-service-ca\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.015074 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.015049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-metrics-certs\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.015074 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.015063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-maas-api-tls\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.020795 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.020775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfhq\" (UniqueName: \"kubernetes.io/projected/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-kube-api-access-czfhq\") pod \"maas-api-6dcd549895-vxcgv\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") " pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.143094 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.142999    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:52.417535 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.417511    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"]
Sep 08 01:01:52.419289 ip-10-0-142-98 kubenswrapper[2555]: W0908 01:01:52.419262    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd536b2c3_1e69_455a_9e1a_378d8a0c2f23.slice/crio-f73b3d4961a5c6ae5f286ce3ab3c1e40199b61b311ee3ba0b6cd2e4d59c98fe4 WatchSource:0}: Error finding container f73b3d4961a5c6ae5f286ce3ab3c1e40199b61b311ee3ba0b6cd2e4d59c98fe4: Status 404 returned error can't find the container with id f73b3d4961a5c6ae5f286ce3ab3c1e40199b61b311ee3ba0b6cd2e4d59c98fe4
Sep 08 01:01:52.731652 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.731570    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:01:52.790362 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:52.790324    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv" event={"ID":"d536b2c3-1e69-455a-9e1a-378d8a0c2f23","Type":"ContainerStarted","Data":"f73b3d4961a5c6ae5f286ce3ab3c1e40199b61b311ee3ba0b6cd2e4d59c98fe4"}
Sep 08 01:01:53.797757 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:53.797719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv" event={"ID":"d536b2c3-1e69-455a-9e1a-378d8a0c2f23","Type":"ContainerStarted","Data":"0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b"}
Sep 08 01:01:53.798152 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:53.797779    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:53.818379 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:53.818330    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv" podStartSLOduration=2.387527905 podStartE2EDuration="2.818316315s" totalImagesPullingTime="430.78841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:01:53.816224015 +0000 UTC m=+1748.159527051" watchObservedRunningTime="2026-09-08 01:01:53.818316315 +0000 UTC m=+1748.161619334"
Sep 08 01:01:59.178713 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.178679    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"]
Sep 08 01:01:59.179099 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.178858    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv" podUID="d536b2c3-1e69-455a-9e1a-378d8a0c2f23" containerName="maas-api" containerID="cri-o://0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b" gracePeriod=30
Sep 08 01:01:59.184899 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.184876    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:59.211421 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.211396    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"]
Sep 08 01:01:59.215068 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.215048    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.222674 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.222653    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"]
Sep 08 01:01:59.379244 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.379212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff3b598-da37-43c3-8617-71fe0091f6ac-maas-api-tls\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.379392 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.379249    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff3b598-da37-43c3-8617-71fe0091f6ac-openshift-service-ca\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.379392 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.379274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff3b598-da37-43c3-8617-71fe0091f6ac-metrics-certs\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.379392 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.379361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5mh\" (UniqueName: \"kubernetes.io/projected/dff3b598-da37-43c3-8617-71fe0091f6ac-kube-api-access-sm5mh\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.422516 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.422488    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:59.480329 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.480247    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff3b598-da37-43c3-8617-71fe0091f6ac-maas-api-tls\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.480329 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.480285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff3b598-da37-43c3-8617-71fe0091f6ac-openshift-service-ca\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.480329 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.480310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff3b598-da37-43c3-8617-71fe0091f6ac-metrics-certs\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.480581 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.480420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5mh\" (UniqueName: \"kubernetes.io/projected/dff3b598-da37-43c3-8617-71fe0091f6ac-kube-api-access-sm5mh\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.481073 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.481050    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff3b598-da37-43c3-8617-71fe0091f6ac-openshift-service-ca\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.482710 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.482691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dff3b598-da37-43c3-8617-71fe0091f6ac-metrics-certs\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.482792 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.482735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dff3b598-da37-43c3-8617-71fe0091f6ac-maas-api-tls\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.488343 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.488322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5mh\" (UniqueName: \"kubernetes.io/projected/dff3b598-da37-43c3-8617-71fe0091f6ac-kube-api-access-sm5mh\") pod \"maas-api-f695cf7ff-gkl9l\" (UID: \"dff3b598-da37-43c3-8617-71fe0091f6ac\") " pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.553429 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.553390    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:01:59.581671 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.581647    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-metrics-certs\") pod \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") "
Sep 08 01:01:59.581774 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.581697    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-maas-api-tls\") pod \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") "
Sep 08 01:01:59.581774 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.581743    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czfhq\" (UniqueName: \"kubernetes.io/projected/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-kube-api-access-czfhq\") pod \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") "
Sep 08 01:01:59.581911 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.581825    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-openshift-service-ca\") pod \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\" (UID: \"d536b2c3-1e69-455a-9e1a-378d8a0c2f23\") "
Sep 08 01:01:59.582298 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.582268    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d536b2c3-1e69-455a-9e1a-378d8a0c2f23" (UID: "d536b2c3-1e69-455a-9e1a-378d8a0c2f23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 01:01:59.583782 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.583751    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d536b2c3-1e69-455a-9e1a-378d8a0c2f23" (UID: "d536b2c3-1e69-455a-9e1a-378d8a0c2f23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 01:01:59.583908 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.583889    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-kube-api-access-czfhq" (OuterVolumeSpecName: "kube-api-access-czfhq") pod "d536b2c3-1e69-455a-9e1a-378d8a0c2f23" (UID: "d536b2c3-1e69-455a-9e1a-378d8a0c2f23"). InnerVolumeSpecName "kube-api-access-czfhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 01:01:59.584246 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.584228    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d536b2c3-1e69-455a-9e1a-378d8a0c2f23" (UID: "d536b2c3-1e69-455a-9e1a-378d8a0c2f23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 01:01:59.683019 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.682990    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-openshift-service-ca\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:59.683019 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.683016    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-metrics-certs\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:59.683019 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.683025    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-maas-api-tls\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:59.683247 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.683035    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czfhq\" (UniqueName: \"kubernetes.io/projected/d536b2c3-1e69-455a-9e1a-378d8a0c2f23-kube-api-access-czfhq\") on node \"ip-10-0-142-98.ec2.internal\" DevicePath \"\""
Sep 08 01:01:59.823895 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.823868    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"]
Sep 08 01:01:59.825534 ip-10-0-142-98 kubenswrapper[2555]: W0908 01:01:59.825505    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff3b598_da37_43c3_8617_71fe0091f6ac.slice/crio-3d93dc202e168f294f8f1513bf3ddafedef29cdcbc63bc823fd5402ce6b11354 WatchSource:0}: Error finding container 3d93dc202e168f294f8f1513bf3ddafedef29cdcbc63bc823fd5402ce6b11354: Status 404 returned error can't find the container with id 3d93dc202e168f294f8f1513bf3ddafedef29cdcbc63bc823fd5402ce6b11354
Sep 08 01:01:59.831014 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.830989    2555 generic.go:356] "Generic (PLEG): container finished" podID="d536b2c3-1e69-455a-9e1a-378d8a0c2f23" containerID="0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b" exitCode=0
Sep 08 01:01:59.831127 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.831026    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"
Sep 08 01:01:59.831127 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.831067    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv" event={"ID":"d536b2c3-1e69-455a-9e1a-378d8a0c2f23","Type":"ContainerDied","Data":"0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b"}
Sep 08 01:01:59.831127 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.831113    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv" event={"ID":"d536b2c3-1e69-455a-9e1a-378d8a0c2f23","Type":"ContainerDied","Data":"f73b3d4961a5c6ae5f286ce3ab3c1e40199b61b311ee3ba0b6cd2e4d59c98fe4"}
Sep 08 01:01:59.831269 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.831137    2555 scope.go:117] "RemoveContainer" containerID="0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b"
Sep 08 01:01:59.847791 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.847766    2555 scope.go:117] "RemoveContainer" containerID="0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b"
Sep 08 01:01:59.848121 ip-10-0-142-98 kubenswrapper[2555]: E0908 01:01:59.848099    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b\": container with ID starting with 0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b not found: ID does not exist" containerID="0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b"
Sep 08 01:01:59.848194 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.848132    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b"} err="failed to get container status \"0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b\": rpc error: code = NotFound desc = could not find container \"0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b\": container with ID starting with 0ea635a3f5d4de5dabb7ced72e87949186176da642c8a52051f548af52131a7b not found: ID does not exist"
Sep 08 01:01:59.858378 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.858349    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"]
Sep 08 01:01:59.862125 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:01:59.862097    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dcd549895-vxcgv"]
Sep 08 01:02:00.155406 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:00.155374    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d536b2c3-1e69-455a-9e1a-378d8a0c2f23" path="/var/lib/kubelet/pods/d536b2c3-1e69-455a-9e1a-378d8a0c2f23/volumes"
Sep 08 01:02:00.839448 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:00.839412    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l" event={"ID":"dff3b598-da37-43c3-8617-71fe0091f6ac","Type":"ContainerStarted","Data":"093ba76188b1b63f98a458dd62cbe722cb3e9e316ab62061512fe7553df1f8a6"}
Sep 08 01:02:00.839931 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:00.839464    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l" event={"ID":"dff3b598-da37-43c3-8617-71fe0091f6ac","Type":"ContainerStarted","Data":"3d93dc202e168f294f8f1513bf3ddafedef29cdcbc63bc823fd5402ce6b11354"}
Sep 08 01:02:00.839931 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:00.839543    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:02:00.859563 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:00.859516    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l" podStartSLOduration=1.553101821 podStartE2EDuration="1.859502836s" totalImagesPullingTime="306.401015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:02:00.857133279 +0000 UTC m=+1755.200436366" watchObservedRunningTime="2026-09-08 01:02:00.859502836 +0000 UTC m=+1755.202806208"
Sep 08 01:02:06.850543 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:06.850515    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f695cf7ff-gkl9l"
Sep 08 01:02:07.433021 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:07.432978    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:02:18.136453 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:18.136417    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:02:30.235868 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:30.235829    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:02:40.038253 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:40.038169    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mpvgw"]
Sep 08 01:02:41.337003 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:41.336968    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784f667-kf4l5_3e41b3fe-7e50-4d2e-a55f-764638857b7f/authorino/0.log"
Sep 08 01:02:46.258841 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:46.258812    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 01:02:46.264397 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:46.264372    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rx52p_724f7c23-3ce3-45dd-932c-c8e0ce407e83/ovn-acl-logging/0.log"
Sep 08 01:02:46.311727 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:46.311698    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-85d49875c6-hfqzv_da04b547-df71-4029-b7ce-1b825d394d50/manager/0.log"
Sep 08 01:02:46.665096 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:46.665055    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dbbcf5b8c-b9glp_158bb49c-6f6e-4e7b-99b4-15e93f86478f/manager/0.log"
Sep 08 01:02:46.887510 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:46.887468    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-bb7b75cc4-w4v82_8ae88f22-e7ce-43c7-80f3-b25681d5f292/manager/0.log"
Sep 08 01:02:47.224120 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:47.224082    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f695cf7ff-gkl9l_dff3b598-da37-43c3-8617-71fe0091f6ac/maas-api/0.log"
Sep 08 01:02:47.555079 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:47.555052    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813820-nzz4p_399337fd-4ea3-4ced-bec5-270b15fd1747/cleanup/0.log"
Sep 08 01:02:48.413010 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:48.412977    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784f667-kf4l5_3e41b3fe-7e50-4d2e-a55f-764638857b7f/authorino/0.log"
Sep 08 01:02:48.523986 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:48.523952    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-84d9bfc694-8kk6x_516d511a-330b-45d4-948c-350aff8791aa/manager/0.log"
Sep 08 01:02:48.631901 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:48.631870    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ffcddf5cd-5r2wd_efa27b1a-738f-461f-af89-1cf92383a7a6/manager/0.log"
Sep 08 01:02:48.737485 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:48.737406    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-hdsmq_e387a42b-e277-4911-a74e-65cad7b7a90c/kuadrant-console-plugin/0.log"
Sep 08 01:02:48.969772 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:48.969744    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-mpvgw_1374a2e4-e8ff-40e7-b603-c9b5d895e079/limitador/0.log"
Sep 08 01:02:49.086272 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:49.086230    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-565895c755-zlh6m_dd1a96b4-1280-4ad0-9f61-92719e3c2c37/manager/0.log"
Sep 08 01:02:49.628226 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:49.628194    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-6xw82_8a24c039-1af6-4761-9f9c-584a6773245c/kube-auth-proxy/0.log"
Sep 08 01:02:50.072068 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:50.072010    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tw9lv_99e916f6-825c-4858-9882-d7670a697d5b/payload-pre-processing/0.log"
Sep 08 01:02:51.057212 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:51.057179    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-c68ddfskg85_100eeba4-5850-4215-acf4-e73d0e7efcdc/main/0.log"
Sep 08 01:02:59.491798 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:59.491762    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fc6wt_7e5f18e2-434c-4068-9d88-d5e4f5f10a8f/global-pull-secret-syncer/0.log"
Sep 08 01:02:59.667557 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:59.667523    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ph5tr_5fea2dc6-69d9-4a32-8d6e-de47f938f884/konnectivity-agent/0.log"
Sep 08 01:02:59.739952 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:02:59.739917    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-98.ec2.internal_4b002b2444c73b6b5500ec2ab14c9652/haproxy/0.log"
Sep 08 01:03:04.661765 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:04.661647    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784f667-kf4l5_3e41b3fe-7e50-4d2e-a55f-764638857b7f/authorino/0.log"
Sep 08 01:03:04.691459 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:04.691424    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-84d9bfc694-8kk6x_516d511a-330b-45d4-948c-350aff8791aa/manager/0.log"
Sep 08 01:03:04.714456 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:04.714424    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ffcddf5cd-5r2wd_efa27b1a-738f-461f-af89-1cf92383a7a6/manager/0.log"
Sep 08 01:03:04.739730 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:04.739699    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-hdsmq_e387a42b-e277-4911-a74e-65cad7b7a90c/kuadrant-console-plugin/0.log"
Sep 08 01:03:04.835056 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:04.835021    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-mpvgw_1374a2e4-e8ff-40e7-b603-c9b5d895e079/limitador/0.log"
Sep 08 01:03:04.901703 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:04.901668    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-565895c755-zlh6m_dd1a96b4-1280-4ad0-9f61-92719e3c2c37/manager/0.log"
Sep 08 01:03:06.673274 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:06.673246    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-8mpj2_b87e8aca-4a77-4c30-b7ba-bdda1c45ad50/cluster-monitoring-operator/0.log"
Sep 08 01:03:06.869710 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:06.869669    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-tdv7m_2625f9b0-0053-4570-a88c-cc2172af60a5/monitoring-plugin/0.log"
Sep 08 01:03:07.102928 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.102899    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pm9p9_5ce59eab-6fe6-45da-b56e-5297ec1da3db/node-exporter/0.log"
Sep 08 01:03:07.125321 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.125234    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pm9p9_5ce59eab-6fe6-45da-b56e-5297ec1da3db/kube-rbac-proxy/0.log"
Sep 08 01:03:07.142301 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.142264    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pm9p9_5ce59eab-6fe6-45da-b56e-5297ec1da3db/init-textfile/0.log"
Sep 08 01:03:07.183072 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.183045    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-g2gms_a0576d64-f3ba-4128-bca9-2ac8ec0190cd/kube-rbac-proxy-main/0.log"
Sep 08 01:03:07.220147 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.220114    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-g2gms_a0576d64-f3ba-4128-bca9-2ac8ec0190cd/kube-rbac-proxy-self/0.log"
Sep 08 01:03:07.246230 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.246202    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-g2gms_a0576d64-f3ba-4128-bca9-2ac8ec0190cd/openshift-state-metrics/0.log"
Sep 08 01:03:07.315783 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.315751    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df/prometheus/0.log"
Sep 08 01:03:07.339272 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.339248    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df/config-reloader/0.log"
Sep 08 01:03:07.360036 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.360012    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df/thanos-sidecar/0.log"
Sep 08 01:03:07.380703 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.380622    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df/kube-rbac-proxy-web/0.log"
Sep 08 01:03:07.401054 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.401031    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df/kube-rbac-proxy/0.log"
Sep 08 01:03:07.425127 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.425101    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df/kube-rbac-proxy-thanos/0.log"
Sep 08 01:03:07.447415 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.447393    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc5905d5-3b4f-48f3-8cd5-32e3d3a9b9df/init-config-reloader/0.log"
Sep 08 01:03:07.526752 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.526722    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-q8k4l_5fae4df9-76bb-45ab-a175-0f821d73db39/prometheus-operator-admission-webhook/0.log"
Sep 08 01:03:07.557171 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.557139    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67fc864b6-ss9cn_b5f6234f-a6e3-4759-9806-ff6844d5a6dd/telemeter-client/0.log"
Sep 08 01:03:07.581029 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.581007    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67fc864b6-ss9cn_b5f6234f-a6e3-4759-9806-ff6844d5a6dd/reload/0.log"
Sep 08 01:03:07.602513 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.602476    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67fc864b6-ss9cn_b5f6234f-a6e3-4759-9806-ff6844d5a6dd/kube-rbac-proxy/0.log"
Sep 08 01:03:07.637810 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.637742    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656dc89847-bqg56_92e2354e-21c3-450f-a060-00fb2141c3a8/thanos-query/0.log"
Sep 08 01:03:07.657830 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.657802    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656dc89847-bqg56_92e2354e-21c3-450f-a060-00fb2141c3a8/kube-rbac-proxy-web/0.log"
Sep 08 01:03:07.678689 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.678661    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656dc89847-bqg56_92e2354e-21c3-450f-a060-00fb2141c3a8/kube-rbac-proxy/0.log"
Sep 08 01:03:07.704580 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.704552    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656dc89847-bqg56_92e2354e-21c3-450f-a060-00fb2141c3a8/prom-label-proxy/0.log"
Sep 08 01:03:07.734853 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.734819    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656dc89847-bqg56_92e2354e-21c3-450f-a060-00fb2141c3a8/kube-rbac-proxy-rules/0.log"
Sep 08 01:03:07.753988 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:07.753962    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656dc89847-bqg56_92e2354e-21c3-450f-a060-00fb2141c3a8/kube-rbac-proxy-metrics/0.log"
Sep 08 01:03:08.912486 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:08.912454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-q72jj_a8ab397c-8640-41f7-a47b-9a9a796412b6/networking-console-plugin/0.log"
Sep 08 01:03:09.223820 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.223743    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"]
Sep 08 01:03:09.224162 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.224149    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d536b2c3-1e69-455a-9e1a-378d8a0c2f23" containerName="maas-api"
Sep 08 01:03:09.224207 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.224163    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d536b2c3-1e69-455a-9e1a-378d8a0c2f23" containerName="maas-api"
Sep 08 01:03:09.224242 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.224223    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d536b2c3-1e69-455a-9e1a-378d8a0c2f23" containerName="maas-api"
Sep 08 01:03:09.227652 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.227636    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.230068 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.230042    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vwnw\"/\"default-dockercfg-ptf2h\""
Sep 08 01:03:09.230068 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.230053    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vwnw\"/\"openshift-service-ca.crt\""
Sep 08 01:03:09.231111 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.231093    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vwnw\"/\"kube-root-ca.crt\""
Sep 08 01:03:09.236178 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.236160    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"]
Sep 08 01:03:09.318260 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.318227    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-proc\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.318260 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.318265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-sys\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.318472 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.318295    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-podres\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.318472 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.318409    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-lib-modules\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.318472 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.318438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvl55\" (UniqueName: \"kubernetes.io/projected/7e065986-5b08-4bdc-a6c1-1776c2588113-kube-api-access-dvl55\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419263 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-lib-modules\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419263 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvl55\" (UniqueName: \"kubernetes.io/projected/7e065986-5b08-4bdc-a6c1-1776c2588113-kube-api-access-dvl55\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419515 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419339    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-proc\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419515 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419370    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-sys\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419515 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-podres\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419515 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-lib-modules\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419515 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-proc\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419515 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419490    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-sys\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.419515 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.419514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e065986-5b08-4bdc-a6c1-1776c2588113-podres\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.426819 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.426791    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvl55\" (UniqueName: \"kubernetes.io/projected/7e065986-5b08-4bdc-a6c1-1776c2588113-kube-api-access-dvl55\") pod \"perf-node-gather-daemonset-q9pbw\" (UID: \"7e065986-5b08-4bdc-a6c1-1776c2588113\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.538915 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.538832    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:09.811224 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:09.811189    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"]
Sep 08 01:03:09.813427 ip-10-0-142-98 kubenswrapper[2555]: W0908 01:03:09.813391    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7e065986_5b08_4bdc_a6c1_1776c2588113.slice/crio-90b5fd266b541fb0b1c7b42c41f81541b00d26c01aae84d6358aa5e15995ce92 WatchSource:0}: Error finding container 90b5fd266b541fb0b1c7b42c41f81541b00d26c01aae84d6358aa5e15995ce92: Status 404 returned error can't find the container with id 90b5fd266b541fb0b1c7b42c41f81541b00d26c01aae84d6358aa5e15995ce92
Sep 08 01:03:10.151044 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:10.150950    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-25kz4_67ed3f60-57cd-4843-96e9-e81b35eae7c5/download-server/0.log"
Sep 08 01:03:10.215179 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:10.215147    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw" event={"ID":"7e065986-5b08-4bdc-a6c1-1776c2588113","Type":"ContainerStarted","Data":"dc1cbf06a89cca88b83e8fdec2fa75ab294e8124693ea121e116952a263fe429"}
Sep 08 01:03:10.215361 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:10.215195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw" event={"ID":"7e065986-5b08-4bdc-a6c1-1776c2588113","Type":"ContainerStarted","Data":"90b5fd266b541fb0b1c7b42c41f81541b00d26c01aae84d6358aa5e15995ce92"}
Sep 08 01:03:10.215361 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:10.215293    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:10.231864 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:10.231815    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw" podStartSLOduration=1.231801566 podStartE2EDuration="1.231801566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:03:10.230817685 +0000 UTC m=+1824.574120705" watchObservedRunningTime="2026-09-08 01:03:10.231801566 +0000 UTC m=+1824.575104584"
Sep 08 01:03:11.812643 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:11.812609    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d9jt5_77df6603-177a-429b-908b-b851cd4e4838/dns/0.log"
Sep 08 01:03:11.832727 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:11.832701    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d9jt5_77df6603-177a-429b-908b-b851cd4e4838/kube-rbac-proxy/0.log"
Sep 08 01:03:12.037784 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:12.037743    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bcfgd_3460b5c1-7a31-4b5a-83df-5ff3ea4eac70/dns-node-resolver/0.log"
Sep 08 01:03:12.661264 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:12.661224    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgjp4_9d01212a-e314-4028-a478-4fd6332e552d/node-ca/0.log"
Sep 08 01:03:14.026077 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:14.026041    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-6xw82_8a24c039-1af6-4761-9f9c-584a6773245c/kube-auth-proxy/0.log"
Sep 08 01:03:14.188912 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:14.188879    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tw9lv_99e916f6-825c-4858-9882-d7670a697d5b/payload-pre-processing/0.log"
Sep 08 01:03:15.094026 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:15.093995    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5d9c4_30a41bc2-2515-4913-8970-b29ef78f764b/serve-healthcheck-canary/0.log"
Sep 08 01:03:15.712768 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:15.712741    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j47nd_b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2/kube-rbac-proxy/0.log"
Sep 08 01:03:15.735037 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:15.735002    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j47nd_b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2/exporter/0.log"
Sep 08 01:03:15.755568 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:15.755536    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j47nd_b91da9ec-a2b0-4c6d-990d-7cfd9a6960b2/extractor/0.log"
Sep 08 01:03:16.230886 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:16.230860    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-q9pbw"
Sep 08 01:03:18.284375 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:18.284347    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c7d4c4566-qtk96_5e788ca2-f7fe-4059-b0b8-1201445767a8/manager/0.log"
Sep 08 01:03:18.350668 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:18.350638    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c7d4c4566-qtk96_5e788ca2-f7fe-4059-b0b8-1201445767a8/copy-manifests/0.log"
Sep 08 01:03:18.955610 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:18.955575    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-6xw82_8a24c039-1af6-4761-9f9c-584a6773245c/kube-auth-proxy/0.log"
Sep 08 01:03:19.082177 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:19.082146    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tw9lv_99e916f6-825c-4858-9882-d7670a697d5b/payload-pre-processing/0.log"
Sep 08 01:03:19.810434 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:19.810404    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-56d489b4b8-tm7n7_e588b11f-7024-4f4b-b7c9-8bc00a857057/jobset-operator/0.log"
Sep 08 01:03:20.455549 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:20.455512    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-85d49875c6-hfqzv_da04b547-df71-4029-b7ce-1b825d394d50/manager/0.log"
Sep 08 01:03:20.758643 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:20.758586    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dbbcf5b8c-b9glp_158bb49c-6f6e-4e7b-99b4-15e93f86478f/manager/0.log"
Sep 08 01:03:20.877471 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:20.877444    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-bb7b75cc4-w4v82_8ae88f22-e7ce-43c7-80f3-b25681d5f292/manager/0.log"
Sep 08 01:03:21.011585 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:21.011501    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-6xw82_8a24c039-1af6-4761-9f9c-584a6773245c/kube-auth-proxy/0.log"
Sep 08 01:03:21.127805 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:21.127770    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-tw9lv_99e916f6-825c-4858-9882-d7670a697d5b/payload-pre-processing/0.log"
Sep 08 01:03:25.689138 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:25.689109    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-djv6g_3a95aabb-fe18-4514-8892-4d43ed3dda07/migrator/0.log"
Sep 08 01:03:25.710130 ip-10-0-142-98 kubenswrapper[2555]: I0908 01:03:25.710094    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-djv6g_3a95aabb-fe18-4514-8892-4d43ed3dda07/graceful-termination/0.log"