Sep 16 02:18:19.404252 ip-10-0-143-48 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 02:18:19.771312 ip-10-0-143-48 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:18:19.771312 ip-10-0-143-48 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 02:18:19.771312 ip-10-0-143-48 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:18:19.771312 ip-10-0-143-48 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 02:18:19.771312 ip-10-0-143-48 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 02:18:19.774239 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.774142    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 02:18:19.778242 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778219    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:19.778242 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778238    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:19.778242 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778242    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:19.778242 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778246    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:19.778242 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778248    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778253    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778258    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778261    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778264    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778267    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778271    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778276    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778279    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778282    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778285    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778287    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778290    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778292    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778295    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778298    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778300    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778303    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778306    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:19.778431 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778308    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778311    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778314    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778316    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778319    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778322    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778325    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778327    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778333    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778335    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778338    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778340    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778343    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778345    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778348    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778351    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778355    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778357    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778360    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778362    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:19.778886 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778365    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778367    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778370    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778373    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778375    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778377    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778380    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778382    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778385    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778387    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778390    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778392    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778395    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778398    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778400    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778403    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778405    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778408    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778410    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778413    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:19.779381 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778415    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778417    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778420    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778423    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778425    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778428    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778430    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778434    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778436    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778439    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778442    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778445    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778448    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778450    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778453    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778456    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778458    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778461    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778463    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778465    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:19.779868 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778468    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778471    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778473    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778476    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778478    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778481    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778484    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778487    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778489    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778491    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778494    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778496    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778500    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778504    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778506    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.778509    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779211    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779218    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779221    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:19.780376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779225    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779229    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779232    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779235    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779237    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779240    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779243    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779246    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779248    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779251    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779254    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779256    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779258    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779261    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779264    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779267    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779270    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779272    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779275    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:19.780832 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779278    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779281    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779283    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779286    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779288    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779292    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779294    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779297    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779299    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779302    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779304    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779308    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779310    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779313    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779316    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779318    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779321    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779323    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779326    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779329    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:19.781313 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779331    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779333    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779336    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779339    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779341    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779344    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779346    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779348    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779352    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779356    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779360    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779363    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779365    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779368    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779371    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779373    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779376    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779381    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779385    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:19.781796 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779388    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779391    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779394    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779397    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779399    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779402    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779405    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779407    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779410    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779412    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779415    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779419    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779421    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779423    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779426    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779429    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779432    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779434    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779437    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779439    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:19.782273 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779442    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779444    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779448    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779451    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779454    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779456    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779459    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779461    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779463    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779466    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779469    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779471    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779474    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779476    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779479    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779481    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779485    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779489    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779888    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:19.782840 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779895    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779899    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779901    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779904    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779907    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779910    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779913    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779917    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779920    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779923    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779940    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779944    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779946    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779949    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779952    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779956    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779959    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779961    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779964    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779967    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:19.783328 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779970    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779973    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779975    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779979    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779984    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779988    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779991    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779994    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.779997    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780000    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780003    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780005    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780008    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780010    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780013    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780015    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780018    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780020    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780024    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:19.783819 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780026    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780029    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780031    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780034    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780036    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780039    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780041    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780044    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780046    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780049    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780052    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780054    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780057    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780060    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780062    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780065    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780067    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780070    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780072    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780075    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:19.784297 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780078    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780080    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780083    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780086    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780088    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780091    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780093    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780096    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780098    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780101    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780103    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780105    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780108    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780113    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780115    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780120    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780123    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780126    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780128    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780131    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:19.784824 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780133    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780136    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780139    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780142    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780144    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780147    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780149    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780151    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780154    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780157    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780159    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780162    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780165    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780167    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780170    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780173    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780175    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780178    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780180    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780254    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 02:18:19.785317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780290    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780296    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780301    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780307    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780310    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780317    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780321    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780324    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780327    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780331    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780334    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780337    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780340    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780343    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780346    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780350    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780353    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780358    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780361    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780364    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780367    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780371    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780374    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 02:18:19.785822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780377    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780380    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780384    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780387    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780390    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780393    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780396    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780399    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780404    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780407    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780410    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780412    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780415    2556 flags.go:64] FLAG: --enable-server="true"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780418    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780424    2556 flags.go:64] FLAG: --event-burst="100"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780428    2556 flags.go:64] FLAG: --event-qps="50"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780430    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780435    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780438    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780442    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780445    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780448    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780451    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780454    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780457    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 02:18:19.786394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780461    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780464    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780467    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780469    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780472    2556 flags.go:64] FLAG: --feature-gates=""
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780476    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780479    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780482    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780485    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780488    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780491    2556 flags.go:64] FLAG: --help="false"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780494    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780497    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780500    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780503    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780506    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780510    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780513    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780516    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780519    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780522    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780526    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780529    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780532    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 16 02:18:19.787019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780536    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780539    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780542    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780545    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780548    2556 flags.go:64] FLAG: --lock-file=""
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780551    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780553    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780557    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780562    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780565    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780568    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780571    2556 flags.go:64] FLAG: --logging-format="text"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780573    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780577    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780579    2556 flags.go:64] FLAG: --manifest-url=""
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780582    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780587    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780590    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780594    2556 flags.go:64] FLAG: --max-pods="110"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780597    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780600    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780603    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780606    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780609    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780612    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 02:18:19.787587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780615    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780622    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780625    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780628    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780633    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780635    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780641    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780644    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780648    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780651    2556 flags.go:64] FLAG: --port="10250"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780654    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780657    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f69ec3eb8df63bf"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780660    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780663    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780666    2556 flags.go:64] FLAG: --register-node="true"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780669    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780672    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780675    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780678    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780681    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780685    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780688    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780691    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780694    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780697    2556 flags.go:64] FLAG: --runonce="false"
Sep 16 02:18:19.788224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780700    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780703    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780706    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780709    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780712    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780715    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780718    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780720    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780723    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780726    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780729    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780734    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780737    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780740    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780742    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780748    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780752    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780754    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780759    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780761    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780764    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780767    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780770    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780773    2556 flags.go:64] FLAG: --v="2"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780777    2556 flags.go:64] FLAG: --version="false"
Sep 16 02:18:19.788831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780782    2556 flags.go:64] FLAG: --vmodule=""
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780785    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.780788    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780900    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780904    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780907    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780910    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780912    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780915    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780918    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780921    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780923    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780939    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780942    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780944    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780949    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780952    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780955    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780960    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:19.789487 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780963    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780966    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780968    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780973    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780976    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780978    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780981    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780984    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780986    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780989    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780992    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780995    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.780997    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781000    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781003    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781006    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781009    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781011    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781014    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:19.789969 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781017    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781021    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781024    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781027    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781030    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781033    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781036    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781039    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781042    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781045    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781047    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781050    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781054    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781057    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781060    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781062    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781066    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781070    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781074    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:19.790439 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781078    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781083    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781087    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781089    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781092    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781094    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781097    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781100    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781103    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781106    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781109    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781111    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781114    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781116    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781119    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781121    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781124    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781127    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781129    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781132    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:19.790902 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781134    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781137    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781140    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781142    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781145    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781149    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781152    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781155    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781157    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781161    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781165    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781168    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781171    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781174    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781176    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781179    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781182    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781185    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781187    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781190    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:19.791403 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781192    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781195    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781198    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781201    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.781203    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.782152    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.789376    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.789392    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.789412    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.789417    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789472    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789477    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789480    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789484    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789488    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:19.791896 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789491    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789494    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789497    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789501    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789504    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789507    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789510    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789513    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789515    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789518    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789522    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789526    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789529    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789532    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789535    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789538    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789541    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789544    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789546    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:19.792305 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789549    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789552    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789555    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789557    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789561    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789567    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789569    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789572    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789575    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789578    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789581    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789583    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789586    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789589    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789591    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789594    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789596    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789599    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789602    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789605    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:19.792802 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789607    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789610    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789612    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789615    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789618    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789620    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789624    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789627    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789630    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789633    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789636    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789639    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789641    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789644    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789647    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789649    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789652    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789656    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789659    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:19.793379 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789662    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789664    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789667    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789670    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789672    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789675    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789678    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789680    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789683    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789685    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789688    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789690    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789693    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789696    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789698    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789700    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789703    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789706    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789708    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789711    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:19.793846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789713    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789716    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789718    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789721    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789724    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789726    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789729    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789731    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789734    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789737    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789740    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789742    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789745    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789748    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789751    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:19.794345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789754    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.789759    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789870    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789876    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789880    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789883    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789886    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789889    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789892    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789895    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789898    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789900    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789903    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789906    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789909    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 02:18:19.794703 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789911    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789914    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789916    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789919    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789922    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789940    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789943    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789946    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789949    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789951    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789954    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789957    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789960    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789963    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789965    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789969    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789972    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789975    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789978    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789980    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 02:18:19.795080 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789983    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789986    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789988    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789991    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789993    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789996    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.789998    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790001    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790005    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790008    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790011    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790014    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790016    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790019    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790022    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790025    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790027    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790030    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790034    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 02:18:19.795562 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790037    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790040    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790043    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790046    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790048    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790051    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790054    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790056    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790059    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790062    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790065    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790067    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790070    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790072    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790075    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790077    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790080    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790082    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790085    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790087    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 02:18:19.796060 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790090    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790092    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790095    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790097    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790100    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790102    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790105    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790107    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790110    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790112    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790115    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790118    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790120    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790123    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790125    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790128    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790131    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790133    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790136    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790138    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 02:18:19.796524 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790141    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790143    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790146    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790149    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790151    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790154    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:19.790157    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.790162    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.790268    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.792217    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.793121    2556 server.go:1019] "Starting client certificate rotation"
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.793215    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.793811    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 02:18:19.797009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.795130    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 02:18:19.799868 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.798565    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 02:18:19.808860 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.808839    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 16 02:18:19.811620 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.811606    2556 log.go:25] "Validated CRI v1 image API"
Sep 16 02:18:19.812557 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.812540    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 02:18:19.815384 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.815366    2556 fs.go:135] Filesystem UUIDs: map[147e0d9f-10f5-43b4-a973-24cbb1c4c326:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b0c85ad9-8a9c-4e6b-86a0-10baf0a8bf8b:/dev/nvme0n1p4]
Sep 16 02:18:19.815453 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.815384    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 02:18:19.821011 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.820887    2556 manager.go:216] Machine: {Timestamp:2026-09-16 02:18:19.819105494 +0000 UTC m=+0.301667436 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122898 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac5c299936eda53ab50d926d61a95 SystemUUID:ec2ac5c2-9993-6eda-53ab-50d926d61a95 BootID:f7e9c6e1-ea4f-4f77-9ab4-3dc2f29e5ec6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:85:2b:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:85:2b:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:31:47:00:c8:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 02:18:19.821011 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.821006    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 02:18:19.821132 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.821087    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 02:18:19.823706 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.823677    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 02:18:19.823847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.823707    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-48.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 02:18:19.824408 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.824394    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 02:18:19.824457 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.824413    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 02:18:19.824457 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.824423    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 02:18:19.825107 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.825094    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 02:18:19.825771 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.825761    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 02:18:19.826480 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.826462    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 02:18:19.828433 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828421    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 02:18:19.828552 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828543    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 02:18:19.828593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828574    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 02:18:19.828593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828586    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 02:18:19.828651 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828596    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 02:18:19.828651 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828606    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 16 02:18:19.828651 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828616    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 02:18:19.828985 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828972    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 02:18:19.829034 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.828993    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 02:18:19.831122 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.831105    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 02:18:19.832805 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.832789    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 02:18:19.832879 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.832828    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 02:18:19.834270 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834256    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 02:18:19.834339 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834277    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 02:18:19.834339 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834285    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 02:18:19.834339 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834292    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 02:18:19.834339 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834304    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 02:18:19.834339 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834325    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 02:18:19.834550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834345    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 02:18:19.834550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834353    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 02:18:19.834550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834364    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 02:18:19.834550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834372    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 02:18:19.834550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834393    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 02:18:19.834550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.834403    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 02:18:19.835348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.835335    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 02:18:19.835434 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.835351    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 02:18:19.838253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.838229    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 02:18:19.838339 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.838271    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 02:18:19.838339 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.838301    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 02:18:19.839114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.839100    2556 server.go:1268] "Started kubelet"
Sep 16 02:18:19.839205 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.839177    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 02:18:19.840096 ip-10-0-143-48 systemd[1]: Started Kubernetes Kubelet.
Sep 16 02:18:19.845125 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.845067    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 02:18:19.845199 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.845149    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 02:18:19.846540 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.846520    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 02:18:19.847309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.847289    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 02:18:19.847521 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.847504    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 02:18:19.851328 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.851308    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 02:18:19.852673 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.852659    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 02:18:19.852814 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.852788    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:19.852952 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.851476    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d48db30d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.839075085 +0000 UTC m=+0.321637027,LastTimestamp:2026-09-16 02:18:19.839075085 +0000 UTC m=+0.321637027,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.853563 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.853546    2556 factory.go:55] Registering systemd factory
Sep 16 02:18:19.853618 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.853566    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 16 02:18:19.853800 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.853788    2556 factory.go:153] Registering CRI-O factory
Sep 16 02:18:19.853846 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.853802    2556 factory.go:223] Registration of the crio container factory successfully
Sep 16 02:18:19.853907 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.853855    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 02:18:19.853907 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.853869    2556 factory.go:103] Registering Raw factory
Sep 16 02:18:19.853907 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.853879    2556 manager.go:1196] Started watching for new ooms in manager
Sep 16 02:18:19.854178 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.854161    2556 manager.go:349] Starting recovery of all containers
Sep 16 02:18:19.854178 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.854168    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 02:18:19.854319 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.854170    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 02:18:19.854319 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.854196    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 02:18:19.854319 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.854302    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 02:18:19.854319 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.854309    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 02:18:19.862000 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.861812    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 02:18:19.862000 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.861887    2556 manager.go:354] Recovery completed
Sep 16 02:18:19.862253 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.862222    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 02:18:19.866729 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.866717    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:19.868594 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.868575    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:19.868663 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.868614    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:19.868663 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.868629    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:19.869227 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.869212    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 02:18:19.869227 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.869226    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 02:18:19.869363 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.869236    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:19.869363 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.869245    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 02:18:19.869908 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.869894    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:19.869985 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.869938    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:19.869985 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.869958    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:19.870295 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.870229    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d6502682  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868595842 +0000 UTC m=+0.351157791,LastTimestamp:2026-09-16 02:18:19.868595842 +0000 UTC m=+0.351157791,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.870451 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.870438    2556 policy_none.go:49] "None policy: Start"
Sep 16 02:18:19.870514 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.870453    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 02:18:19.870514 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.870464    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 02:18:19.871390 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.871378    2556 policy_none.go:47] "Start"
Sep 16 02:18:19.879024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.878999    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2brzx"
Sep 16 02:18:19.880921 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.880853    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d6508ad3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868621523 +0000 UTC m=+0.351183469,LastTimestamp:2026-09-16 02:18:19.868621523 +0000 UTC m=+0.351183469,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.886172 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.886154    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2brzx"
Sep 16 02:18:19.891620 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.891555    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d650bc40  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868634176 +0000 UTC m=+0.351196119,LastTimestamp:2026-09-16 02:18:19.868634176 +0000 UTC m=+0.351196119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.897632 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.897617    2556 manager.go:337] "Starting Device Plugin manager"
Sep 16 02:18:19.897722 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.897662    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 02:18:19.897722 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.897675    2556 server.go:87] "Starting device plugin registration server"
Sep 16 02:18:19.898088 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.898071    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 02:18:19.898192 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.898091    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 02:18:19.898267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.898208    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 02:18:19.898332 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.898290    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 02:18:19.898332 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.898298    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 02:18:19.898884 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.898865    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 02:18:19.898971 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.898909    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:19.899741 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.899673    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-48.ec2.internal.18d5ab98d6502682\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d6502682  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868595842 +0000 UTC m=+0.351157791,LastTimestamp:2026-09-16 02:18:19.869908724 +0000 UTC m=+0.352470682,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.907785 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.907704    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-48.ec2.internal.18d5ab98d6508ad3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d6508ad3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868621523 +0000 UTC m=+0.351183469,LastTimestamp:2026-09-16 02:18:19.869945258 +0000 UTC m=+0.352507203,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.918727 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.918644    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-48.ec2.internal.18d5ab98d650bc40\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d650bc40  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868634176 +0000 UTC m=+0.351196119,LastTimestamp:2026-09-16 02:18:19.869964007 +0000 UTC m=+0.352525949,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.933535 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:19.927409    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d83708ff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.900504319 +0000 UTC m=+0.383066248,LastTimestamp:2026-09-16 02:18:19.900504319 +0000 UTC m=+0.383066248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:19.998836 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:19.998807    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:20.000769 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.000753    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:20.000849 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.000783    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:20.000849 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.000794    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:20.000849 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.000820    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.009565 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.009489    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-48.ec2.internal.18d5ab98d6502682\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d6502682  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868595842 +0000 UTC m=+0.351157791,LastTimestamp:2026-09-16 02:18:20.000769084 +0000 UTC m=+0.483331036,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:20.012763 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.012739    2556 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.012837 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.012754    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-48.ec2.internal.18d5ab98d6508ad3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d6508ad3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868621523 +0000 UTC m=+0.351183469,LastTimestamp:2026-09-16 02:18:20.000787259 +0000 UTC m=+0.483349212,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:20.016504 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.016445    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-48.ec2.internal.18d5ab98d650bc40\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.ec2.internal.18d5ab98d650bc40  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.ec2.internal,UID:ip-10-0-143-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.ec2.internal,},FirstTimestamp:2026-09-16 02:18:19.868634176 +0000 UTC m=+0.351196119,LastTimestamp:2026-09-16 02:18:20.000797927 +0000 UTC m=+0.483359869,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.ec2.internal,}"
Sep 16 02:18:20.053092 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.052996    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 02:18:20.054253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.054238    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 02:18:20.054313 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.054288    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 02:18:20.054313 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.054311    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 02:18:20.054420 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.054406    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 02:18:20.056212 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.056187    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 16 02:18:20.077129 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.077107    2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-48.ec2.internal\" not found" node="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.154531 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.154502    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"]
Sep 16 02:18:20.154627 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.154579    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:20.155621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.155606    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:20.155743 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.155633    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:20.155743 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.155643    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:20.157044 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157031    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:20.157173 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.157215 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157193    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:20.157819 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157801    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:20.157882 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157830    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:20.157882 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157839    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:20.157882 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157802    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:20.157991 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157895    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:20.157991 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.157909    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:20.159482 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.159467    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.159563 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.159490    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:20.160233 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.160219    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:20.160298 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.160246    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:20.160298 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.160260    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:20.180733 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.180708    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-48.ec2.internal\" not found" node="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.185115 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.185098    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-48.ec2.internal\" not found" node="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.212823 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.212804    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:20.213749 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.213732    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:20.213846 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.213765    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:20.213846 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.213795    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:20.213846 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.213826    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.221644 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.221626    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.221729 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.221653    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-48.ec2.internal\": node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:20.248151 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.248122    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:20.254917 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.254892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d37433c425db0f461fce4d25fa7def78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"d37433c425db0f461fce4d25fa7def78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.255046 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.254948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d37433c425db0f461fce4d25fa7def78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"d37433c425db0f461fce4d25fa7def78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.255046 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.254976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c5e07eb16b748b05b653e47f2fc353a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.ec2.internal\" (UID: \"c5e07eb16b748b05b653e47f2fc353a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.348408 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:20.348323    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:20.355693 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.355667    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d37433c425db0f461fce4d25fa7def78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"d37433c425db0f461fce4d25fa7def78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.355758 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.355699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c5e07eb16b748b05b653e47f2fc353a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.ec2.internal\" (UID: \"c5e07eb16b748b05b653e47f2fc353a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.355758 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.355714    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d37433c425db0f461fce4d25fa7def78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"d37433c425db0f461fce4d25fa7def78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.355838 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.355756    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d37433c425db0f461fce4d25fa7def78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"d37433c425db0f461fce4d25fa7def78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.355838 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.355774    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d37433c425db0f461fce4d25fa7def78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal\" (UID: \"d37433c425db0f461fce4d25fa7def78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:20.355838 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:20.355778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c5e07eb16b748b05b653e47f2fc353a6-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.ec2.internal\" (UID: \"c5e07eb16b748b05b653e47f2fc353a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:21.501369 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:21.501331    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:21.535893 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:21.535859    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:21.540614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:21.540594    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:21.602266 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:21.602227    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:21.702882 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:21.702808    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:21.803413 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:21.803376    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:21.845050 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:21.845025    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 02:18:21.903589 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:21.903561    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:22.004303 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:22.004217    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:22.054709 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.054678    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 02:18:22.056078 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.056052    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 02:18:22.056209 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.056101    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 02:18:22.056209 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.056115    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 02:18:22.065246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.065223    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:22.071061 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.071003    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 02:13:20 +0000 UTC" deadline="2028-06-24 01:24:07.078118378 +0000 UTC"
Sep 16 02:18:22.071148 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.071061    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15527h5m45.007061497s"
Sep 16 02:18:22.105210 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:22.105179    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:22.178373 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:22.178333    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e07eb16b748b05b653e47f2fc353a6.slice/crio-a78c8586efbc8e2bbaaaf56fbd5f95abdd7fde30d3726af2006a0eb1423d8dc6 WatchSource:0}: Error finding container a78c8586efbc8e2bbaaaf56fbd5f95abdd7fde30d3726af2006a0eb1423d8dc6: Status 404 returned error can't find the container with id a78c8586efbc8e2bbaaaf56fbd5f95abdd7fde30d3726af2006a0eb1423d8dc6
Sep 16 02:18:22.178799 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:22.178775    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37433c425db0f461fce4d25fa7def78.slice/crio-c6136a1b235aa8489e3bd55980c65e8d3e3e49cd9268dcf3b20acce93b355ae9 WatchSource:0}: Error finding container c6136a1b235aa8489e3bd55980c65e8d3e3e49cd9268dcf3b20acce93b355ae9: Status 404 returned error can't find the container with id c6136a1b235aa8489e3bd55980c65e8d3e3e49cd9268dcf3b20acce93b355ae9
Sep 16 02:18:22.184158 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.184142    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:18:22.205474 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:22.205419    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-48.ec2.internal\" not found"
Sep 16 02:18:22.230560 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.230535    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:22.305416 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.305349    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:22.316632 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.316609    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 02:18:22.318273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.318259    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal"
Sep 16 02:18:22.323833 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.323816    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 02:18:22.367979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.366661    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:22.404767 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.404740    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 02:18:22.881377 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.881342    2556 apiserver.go:52] "Watching apiserver"
Sep 16 02:18:22.888038 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.888002    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 02:18:22.888585 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.888560    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-plfnq","kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj","openshift-dns/node-resolver-jn7jj","openshift-multus/multus-additional-cni-plugins-gdff9","openshift-multus/network-metrics-daemon-t4vxs","openshift-network-operator/iptables-alerter-5q6nd","openshift-ovn-kubernetes/ovnkube-node-z8qvd","kube-system/konnectivity-agent-tpdvn","openshift-cluster-node-tuning-operator/tuned-rd5kz","openshift-image-registry/node-ca-9w9ds","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal","openshift-multus/multus-bzlfw","openshift-network-diagnostics/network-check-target-4sfbv"]
Sep 16 02:18:22.891438 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.891401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:22.893460 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.893423    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.894205 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.894185    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:22.894298 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.894282    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 02:18:22.894481 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.894464    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:22.894554 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.894512    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nb2cv\""
Sep 16 02:18:22.895637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.895617    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 02:18:22.895732 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.895657    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 02:18:22.895818 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.895801    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nkq6s\""
Sep 16 02:18:22.895940 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.895923    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.896008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.895969    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 02:18:22.896143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.895931    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 02:18:22.898313 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.898293    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 02:18:22.898401 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.898333    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 02:18:22.898527 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.898512    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:22.898637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.898618    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sv6k9\""
Sep 16 02:18:22.898917 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.898896    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 02:18:22.900678 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.900657    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 02:18:22.900916 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.900898    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 02:18:22.901214 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.901194    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xb5rd\""
Sep 16 02:18:22.903802 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.902775    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.904694 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.904673    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 02:18:22.905381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.904870    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:22.905381 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:22.904945    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:22.905381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.904995    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 02:18:22.905381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.905200    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 02:18:22.905381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.905236    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5z8vk\""
Sep 16 02:18:22.907419 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.907072    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:22.909377 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.909358    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:22.909471 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:22.909450    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:22.909860 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.909841    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 02:18:22.910768 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.910416    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k5js4\""
Sep 16 02:18:22.910768 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.910351    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 02:18:22.911755 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.911645    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:22.911755 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:22.911706    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:22.914002 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.913983    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:22.914994 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.914974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-multus-certs\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.915199 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915180    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-etc-kubernetes\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.915323 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915308    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.915683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915408    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-system-cni-dir\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.915683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915455    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.915683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915481    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-k8s-cni-cncf-io\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.915683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ea3ca72-53a1-456c-85f1-8750d3ac07b1-agent-certs\") pod \"konnectivity-agent-tpdvn\" (UID: \"8ea3ca72-53a1-456c-85f1-8750d3ac07b1\") " pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:22.915683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-socket-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.915683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-device-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.915683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.915573    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-sys-fs\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b5ea2d3-a6ae-4089-8204-d7422983fe38-hosts-file\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916172    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn64h\" (UniqueName: \"kubernetes.io/projected/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-kube-api-access-dn64h\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916688    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-cni-binary-copy\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916712    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-socket-dir-parent\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916735    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916829    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-registration-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916866    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d5ac133-a00e-48e5-94d6-240cafe853ad-iptables-alerter-script\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916913    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916947    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-netns\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.916970    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-hostroot\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-daemon-config\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917033    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7k5q\" (UniqueName: \"kubernetes.io/projected/e2c8009a-e055-450a-9175-53a0d5291656-kube-api-access-f7k5q\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-cnibin\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.917114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-system-cni-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-conf-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5ea2d3-a6ae-4089-8204-d7422983fe38-tmp-dir\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917179    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ea3ca72-53a1-456c-85f1-8750d3ac07b1-konnectivity-ca\") pod \"konnectivity-agent-tpdvn\" (UID: \"8ea3ca72-53a1-456c-85f1-8750d3ac07b1\") " pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5ac133-a00e-48e5-94d6-240cafe853ad-host-slash\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z47r\" (UniqueName: \"kubernetes.io/projected/2d5ac133-a00e-48e5-94d6-240cafe853ad-kube-api-access-7z47r\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917274    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-cnibin\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917297    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpl8c\" (UniqueName: \"kubernetes.io/projected/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-kube-api-access-lpl8c\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917325    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pff5\" (UniqueName: \"kubernetes.io/projected/5b5ea2d3-a6ae-4089-8204-d7422983fe38-kube-api-access-9pff5\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917363    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.917815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.917391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-cni-multus\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.919813 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.919781    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.919911 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.919846    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-cni-bin\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.919911 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.919888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-kubelet\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.920010 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.919915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpfw\" (UniqueName: \"kubernetes.io/projected/2c014add-556b-46aa-a326-69b9bad2ed52-kube-api-access-7fpfw\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:22.920010 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.919946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-cni-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:22.920363 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.920339    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:22.920467 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.920449    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:22.920535 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.920514    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 02:18:22.921028 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.921005    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 02:18:22.921116 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.921098    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4wvg9\""
Sep 16 02:18:22.921500 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.921476    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 02:18:22.921587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.921501    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 02:18:22.923248 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.921719    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p94bc\""
Sep 16 02:18:22.923248 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.921894    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:22.924527 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.924093    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 02:18:22.925515 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.924808    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 02:18:22.925515 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.925105    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 02:18:22.925515 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.925411    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 02:18:22.925757 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.925584    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cb98\""
Sep 16 02:18:22.925757 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.925681    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 02:18:22.925847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.925811    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 02:18:22.935935 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.935908    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vb6s4"
Sep 16 02:18:22.943477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:22.943451    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vb6s4"
Sep 16 02:18:23.007866 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.007832    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 02:18:23.020180 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020136    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-system-cni-dir\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-k8s-cni-cncf-io\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020225    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-system-cni-dir\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020244    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-socket-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020284    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-device-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020304    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b5ea2d3-a6ae-4089-8204-d7422983fe38-hosts-file\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020328    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-socket-dir-parent\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b83e697-837d-4c0e-b454-767d6743f7ab-kubelet-config\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020366    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68c43980-26d5-4302-bdc8-127d3fbdffeb-tmp\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.020397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020382    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg94t\" (UniqueName: \"kubernetes.io/projected/6e60ad12-0057-402f-b9fb-56ffa04e13a0-kube-api-access-hg94t\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020412    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-socket-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-registration-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d5ac133-a00e-48e5-94d6-240cafe853ad-iptables-alerter-script\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020476    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysctl-conf\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020485    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-k8s-cni-cncf-io\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020498    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-systemd\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020525    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-var-lib-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020542    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-etc-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020550    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-device-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-tuned\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020595    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97mnh\" (UniqueName: \"kubernetes.io/projected/68c43980-26d5-4302-bdc8-127d3fbdffeb-kube-api-access-97mnh\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b5ea2d3-a6ae-4089-8204-d7422983fe38-hosts-file\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-kubelet\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020647    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-socket-dir-parent\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.020890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ea3ca72-53a1-456c-85f1-8750d3ac07b1-konnectivity-ca\") pod \"konnectivity-agent-tpdvn\" (UID: \"8ea3ca72-53a1-456c-85f1-8750d3ac07b1\") " pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5ac133-a00e-48e5-94d6-240cafe853ad-host-slash\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysconfig\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020781    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020812    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60ad12-0057-402f-b9fb-56ffa04e13a0-serviceca\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020861    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d5ac133-a00e-48e5-94d6-240cafe853ad-host-slash\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020866    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/725b636c-96b2-4291-9f13-6133cf5ce923-ovn-node-metrics-cert\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020897    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pff5\" (UniqueName: \"kubernetes.io/projected/5b5ea2d3-a6ae-4089-8204-d7422983fe38-kube-api-access-9pff5\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-registration-dir\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020921    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-cni-multus\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.020973    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-cni-multus\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021001    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b83e697-837d-4c0e-b454-767d6743f7ab-dbus\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.021022    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-kubernetes\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.021098    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs podName:103d25bf-1d27-4cb9-be5c-cf6a8f8e691b nodeName:}" failed. No retries permitted until 2026-09-16 02:18:23.521074827 +0000 UTC m=+2.951136209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs") pod "network-metrics-daemon-t4vxs" (UID: "103d25bf-1d27-4cb9-be5c-cf6a8f8e691b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021163    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d5ac133-a00e-48e5-94d6-240cafe853ad-iptables-alerter-script\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:23.021660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021163    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ea3ca72-53a1-456c-85f1-8750d3ac07b1-konnectivity-ca\") pod \"konnectivity-agent-tpdvn\" (UID: \"8ea3ca72-53a1-456c-85f1-8750d3ac07b1\") " pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021242    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-cni-bin\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-node-log\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021317    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-ovnkube-script-lib\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-cni-bin\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021370    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpfw\" (UniqueName: \"kubernetes.io/projected/2c014add-556b-46aa-a326-69b9bad2ed52-kube-api-access-7fpfw\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021397    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-multus-certs\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021584    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-multus-certs\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021623    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-etc-kubernetes\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021650    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-systemd-units\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021703    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-etc-kubernetes\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021731    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021901    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysctl-d\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ea3ca72-53a1-456c-85f1-8750d3ac07b1-agent-certs\") pod \"konnectivity-agent-tpdvn\" (UID: \"8ea3ca72-53a1-456c-85f1-8750d3ac07b1\") " pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:23.022477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.021991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-sys-fs\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn64h\" (UniqueName: \"kubernetes.io/projected/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-kube-api-access-dn64h\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022047    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-cni-binary-copy\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-netns\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022099    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2c8009a-e055-450a-9175-53a0d5291656-sys-fs\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-var-lib-kubelet\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022140    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-cni-netd\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022172    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-hostroot\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022214    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-daemon-config\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-modprobe-d\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022253    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-systemd\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022272    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-env-overrides\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7k5q\" (UniqueName: \"kubernetes.io/projected/e2c8009a-e055-450a-9175-53a0d5291656-kube-api-access-f7k5q\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-cnibin\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022356    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-system-cni-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022366    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.023240 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-conf-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60ad12-0057-402f-b9fb-56ffa04e13a0-host\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-hostroot\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022513    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-run-netns\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022660    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-cnibin\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022719    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbfs\" (UniqueName: \"kubernetes.io/projected/725b636c-96b2-4291-9f13-6133cf5ce923-kube-api-access-cgbfs\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022710    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022807    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022953    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-cni-binary-copy\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.022973    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-daemon-config\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-conf-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5ea2d3-a6ae-4089-8204-d7422983fe38-tmp-dir\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z47r\" (UniqueName: \"kubernetes.io/projected/2d5ac133-a00e-48e5-94d6-240cafe853ad-kube-api-access-7z47r\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023283    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-system-cni-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-cnibin\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpl8c\" (UniqueName: \"kubernetes.io/projected/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-kube-api-access-lpl8c\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b5ea2d3-a6ae-4089-8204-d7422983fe38-tmp-dir\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023483    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-cnibin\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.023959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023487    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-run\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023562    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-ovn\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-sys\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-lib-modules\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023677    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-host\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-slash\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023739    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023760    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-kubelet\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-run-netns\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023800    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-host-var-lib-kubelet\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023811    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c014add-556b-46aa-a326-69b9bad2ed52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-log-socket\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023844    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023863    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-cni-bin\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023901    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-cni-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.024652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023936    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-ovnkube-config\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.025273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.023999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-multus-cni-dir\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.025273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.024176    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c014add-556b-46aa-a326-69b9bad2ed52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.026561 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.026539    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ea3ca72-53a1-456c-85f1-8750d3ac07b1-agent-certs\") pod \"konnectivity-agent-tpdvn\" (UID: \"8ea3ca72-53a1-456c-85f1-8750d3ac07b1\") " pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:23.035131 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.035094    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pff5\" (UniqueName: \"kubernetes.io/projected/5b5ea2d3-a6ae-4089-8204-d7422983fe38-kube-api-access-9pff5\") pod \"node-resolver-jn7jj\" (UID: \"5b5ea2d3-a6ae-4089-8204-d7422983fe38\") " pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:23.036125 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.036097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn64h\" (UniqueName: \"kubernetes.io/projected/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-kube-api-access-dn64h\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:23.036249 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.036150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z47r\" (UniqueName: \"kubernetes.io/projected/2d5ac133-a00e-48e5-94d6-240cafe853ad-kube-api-access-7z47r\") pod \"iptables-alerter-5q6nd\" (UID: \"2d5ac133-a00e-48e5-94d6-240cafe853ad\") " pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:23.036895 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.036879    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpl8c\" (UniqueName: \"kubernetes.io/projected/7ab38bf5-ae74-4023-8c82-d79ce7db57dd-kube-api-access-lpl8c\") pod \"multus-bzlfw\" (UID: \"7ab38bf5-ae74-4023-8c82-d79ce7db57dd\") " pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.036958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.036903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpfw\" (UniqueName: \"kubernetes.io/projected/2c014add-556b-46aa-a326-69b9bad2ed52-kube-api-access-7fpfw\") pod \"multus-additional-cni-plugins-gdff9\" (UID: \"2c014add-556b-46aa-a326-69b9bad2ed52\") " pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.037360 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.037341    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7k5q\" (UniqueName: \"kubernetes.io/projected/e2c8009a-e055-450a-9175-53a0d5291656-kube-api-access-f7k5q\") pod \"aws-ebs-csi-driver-node-7cwzj\" (UID: \"e2c8009a-e055-450a-9175-53a0d5291656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.112722 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.112664    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal" event={"ID":"c5e07eb16b748b05b653e47f2fc353a6","Type":"ContainerStarted","Data":"a78c8586efbc8e2bbaaaf56fbd5f95abdd7fde30d3726af2006a0eb1423d8dc6"}
Sep 16 02:18:23.113974 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.113938    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" event={"ID":"d37433c425db0f461fce4d25fa7def78","Type":"ContainerStarted","Data":"c6136a1b235aa8489e3bd55980c65e8d3e3e49cd9268dcf3b20acce93b355ae9"}
Sep 16 02:18:23.124387 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60ad12-0057-402f-b9fb-56ffa04e13a0-host\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.124387 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbfs\" (UniqueName: \"kubernetes.io/projected/725b636c-96b2-4291-9f13-6133cf5ce923-kube-api-access-cgbfs\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-run\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124448    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-ovn\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124446    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e60ad12-0057-402f-b9fb-56ffa04e13a0-host\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-sys\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124493    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-lib-modules\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124515    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-host\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124527    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-run\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124541    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-slash\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124565    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-run-netns\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124570    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-sys\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124585    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-log-socket\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124595    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-ovn\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124607    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124623    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-slash\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.124631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124629    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-cni-bin\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124653    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-ovnkube-config\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124701    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-lib-modules\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124727    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b83e697-837d-4c0e-b454-767d6743f7ab-kubelet-config\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124726    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-log-socket\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-run-netns\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68c43980-26d5-4302-bdc8-127d3fbdffeb-tmp\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124773    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg94t\" (UniqueName: \"kubernetes.io/projected/6e60ad12-0057-402f-b9fb-56ffa04e13a0-kube-api-access-hg94t\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124780    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-host\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124785    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-cni-bin\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysctl-conf\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124820    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b83e697-837d-4c0e-b454-767d6743f7ab-kubelet-config\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-systemd\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.125287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124865    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-var-lib-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124883    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-systemd\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-var-lib-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124924    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysctl-conf\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124886    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-etc-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124924    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-etc-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124954    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.124978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-tuned\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125053    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-openvswitch\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125085    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97mnh\" (UniqueName: \"kubernetes.io/projected/68c43980-26d5-4302-bdc8-127d3fbdffeb-kube-api-access-97mnh\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-kubelet\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125147    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysconfig\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125171    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60ad12-0057-402f-b9fb-56ffa04e13a0-serviceca\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-kubelet\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/725b636c-96b2-4291-9f13-6133cf5ce923-ovn-node-metrics-cert\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b83e697-837d-4c0e-b454-767d6743f7ab-dbus\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125278    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-kubernetes\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-ovnkube-config\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125296    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysconfig\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-node-log\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-ovnkube-script-lib\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125383    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-node-log\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-systemd-units\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125422    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-systemd-units\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125464    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b83e697-837d-4c0e-b454-767d6743f7ab-dbus\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysctl-d\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-var-lib-kubelet\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.125517    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-cni-netd\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.125589    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret podName:9b83e697-837d-4c0e-b454-767d6743f7ab nodeName:}" failed. No retries permitted until 2026-09-16 02:18:23.625571142 +0000 UTC m=+3.055632520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret") pod "global-pull-secret-syncer-plfnq" (UID: "9b83e697-837d-4c0e-b454-767d6743f7ab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-modprobe-d\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-systemd\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125659    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-env-overrides\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125744    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e60ad12-0057-402f-b9fb-56ffa04e13a0-serviceca\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.126938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125783    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-var-lib-kubelet\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.127783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125354    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-kubernetes\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.127783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125824    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-host-cni-netd\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.127783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125857    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/725b636c-96b2-4291-9f13-6133cf5ce923-run-systemd\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.127783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125875    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-ovnkube-script-lib\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.127783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125891    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-modprobe-d\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.127783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.125934    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-sysctl-d\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.127783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.126113    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/725b636c-96b2-4291-9f13-6133cf5ce923-env-overrides\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.128140 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.127954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68c43980-26d5-4302-bdc8-127d3fbdffeb-tmp\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.128140 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.128062    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68c43980-26d5-4302-bdc8-127d3fbdffeb-etc-tuned\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.128703 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.128294    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/725b636c-96b2-4291-9f13-6133cf5ce923-ovn-node-metrics-cert\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.131291 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.131269    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:23.131291 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.131292    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:23.131491 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.131305    2556 projected.go:196] Error preparing data for projected volume kube-api-access-62w8z for pod openshift-network-diagnostics/network-check-target-4sfbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:23.131491 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.131366    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z podName:3b94c46a-de62-4b93-ba0c-66059c365abb nodeName:}" failed. No retries permitted until 2026-09-16 02:18:23.63134764 +0000 UTC m=+3.061409015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-62w8z" (UniqueName: "kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z") pod "network-check-target-4sfbv" (UID: "3b94c46a-de62-4b93-ba0c-66059c365abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:23.133698 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.133667    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbfs\" (UniqueName: \"kubernetes.io/projected/725b636c-96b2-4291-9f13-6133cf5ce923-kube-api-access-cgbfs\") pod \"ovnkube-node-z8qvd\" (UID: \"725b636c-96b2-4291-9f13-6133cf5ce923\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.134027 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.133999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97mnh\" (UniqueName: \"kubernetes.io/projected/68c43980-26d5-4302-bdc8-127d3fbdffeb-kube-api-access-97mnh\") pod \"tuned-rd5kz\" (UID: \"68c43980-26d5-4302-bdc8-127d3fbdffeb\") " pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.134295 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.134276    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg94t\" (UniqueName: \"kubernetes.io/projected/6e60ad12-0057-402f-b9fb-56ffa04e13a0-kube-api-access-hg94t\") pod \"node-ca-9w9ds\" (UID: \"6e60ad12-0057-402f-b9fb-56ffa04e13a0\") " pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.208372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.208330    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5q6nd"
Sep 16 02:18:23.221331 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.221299    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bzlfw"
Sep 16 02:18:23.233808 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.233767    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj"
Sep 16 02:18:23.244649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.244619    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jn7jj"
Sep 16 02:18:23.252509 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.252481    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gdff9"
Sep 16 02:18:23.257282 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.257261    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:23.263903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.263883    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rd5kz"
Sep 16 02:18:23.274560 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.274530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9w9ds"
Sep 16 02:18:23.280492 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.280464    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:23.528805 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.528711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:23.528962 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.528849    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:23.528962 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.528911    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs podName:103d25bf-1d27-4cb9-be5c-cf6a8f8e691b nodeName:}" failed. No retries permitted until 2026-09-16 02:18:24.528894241 +0000 UTC m=+3.958955621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs") pod "network-metrics-daemon-t4vxs" (UID: "103d25bf-1d27-4cb9-be5c-cf6a8f8e691b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:23.629749 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.629712    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:23.629944 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.629887    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:23.629996 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.629973    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret podName:9b83e697-837d-4c0e-b454-767d6743f7ab nodeName:}" failed. No retries permitted until 2026-09-16 02:18:24.629952208 +0000 UTC m=+4.060013593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret") pod "global-pull-secret-syncer-plfnq" (UID: "9b83e697-837d-4c0e-b454-767d6743f7ab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:23.730784 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.730749    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:23.730965 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.730945    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:23.731024 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.730972    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:23.731024 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.730985    2556 projected.go:196] Error preparing data for projected volume kube-api-access-62w8z for pod openshift-network-diagnostics/network-check-target-4sfbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:23.731100 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:23.731057    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z podName:3b94c46a-de62-4b93-ba0c-66059c365abb nodeName:}" failed. No retries permitted until 2026-09-16 02:18:24.731039683 +0000 UTC m=+4.161101068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-62w8z" (UniqueName: "kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z") pod "network-check-target-4sfbv" (UID: "3b94c46a-de62-4b93-ba0c-66059c365abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:23.946606 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.946514    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 02:13:22 +0000 UTC" deadline="2028-05-13 00:54:06.538034501 +0000 UTC"
Sep 16 02:18:23.946606 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:23.946555    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14518h35m42.591484485s"
Sep 16 02:18:24.162002 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.161724    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e60ad12_0057_402f_b9fb_56ffa04e13a0.slice/crio-0ec48270ac6fedd809a8fb75b9fc69ead0239dcd0ae0a2b05f58eeb3779abe83 WatchSource:0}: Error finding container 0ec48270ac6fedd809a8fb75b9fc69ead0239dcd0ae0a2b05f58eeb3779abe83: Status 404 returned error can't find the container with id 0ec48270ac6fedd809a8fb75b9fc69ead0239dcd0ae0a2b05f58eeb3779abe83
Sep 16 02:18:24.164066 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.163814    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c014add_556b_46aa_a326_69b9bad2ed52.slice/crio-a1f730555604db953c09ad25908ce20208ecb8c4fd1552caca89bb3b422c6f18 WatchSource:0}: Error finding container a1f730555604db953c09ad25908ce20208ecb8c4fd1552caca89bb3b422c6f18: Status 404 returned error can't find the container with id a1f730555604db953c09ad25908ce20208ecb8c4fd1552caca89bb3b422c6f18
Sep 16 02:18:24.167159 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.167140    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab38bf5_ae74_4023_8c82_d79ce7db57dd.slice/crio-36f873ee9785329709da6787f38732a318fce5e1ca842e0cb5d644f973723048 WatchSource:0}: Error finding container 36f873ee9785329709da6787f38732a318fce5e1ca842e0cb5d644f973723048: Status 404 returned error can't find the container with id 36f873ee9785329709da6787f38732a318fce5e1ca842e0cb5d644f973723048
Sep 16 02:18:24.168347 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.168090    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c43980_26d5_4302_bdc8_127d3fbdffeb.slice/crio-361204d6b569455994cdaa6e7878d234c3e01305f03c94f882b1d7c93c626b60 WatchSource:0}: Error finding container 361204d6b569455994cdaa6e7878d234c3e01305f03c94f882b1d7c93c626b60: Status 404 returned error can't find the container with id 361204d6b569455994cdaa6e7878d234c3e01305f03c94f882b1d7c93c626b60
Sep 16 02:18:24.169132 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.169106    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725b636c_96b2_4291_9f13_6133cf5ce923.slice/crio-e66836c888e2c20c0afe6be89ce2aa35c7a70c21eea9b4c07b9469020abfc959 WatchSource:0}: Error finding container e66836c888e2c20c0afe6be89ce2aa35c7a70c21eea9b4c07b9469020abfc959: Status 404 returned error can't find the container with id e66836c888e2c20c0afe6be89ce2aa35c7a70c21eea9b4c07b9469020abfc959
Sep 16 02:18:24.170188 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.170175    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5ac133_a00e_48e5_94d6_240cafe853ad.slice/crio-8da56e3aaaab7322de0b276241ce45bec728f69f56975a10ede3dc4c77770739 WatchSource:0}: Error finding container 8da56e3aaaab7322de0b276241ce45bec728f69f56975a10ede3dc4c77770739: Status 404 returned error can't find the container with id 8da56e3aaaab7322de0b276241ce45bec728f69f56975a10ede3dc4c77770739
Sep 16 02:18:24.171229 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.171098    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea3ca72_53a1_456c_85f1_8750d3ac07b1.slice/crio-e6b28c5c8b344be291f23039be3e8cbfd36fb27bee1614e7caad93bf2db3ada1 WatchSource:0}: Error finding container e6b28c5c8b344be291f23039be3e8cbfd36fb27bee1614e7caad93bf2db3ada1: Status 404 returned error can't find the container with id e6b28c5c8b344be291f23039be3e8cbfd36fb27bee1614e7caad93bf2db3ada1
Sep 16 02:18:24.172626 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.172592    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c8009a_e055_450a_9175_53a0d5291656.slice/crio-5ac671622017f7e1d511956fff1d1ba9744c20903ac7782f1dd5baa52495d64c WatchSource:0}: Error finding container 5ac671622017f7e1d511956fff1d1ba9744c20903ac7782f1dd5baa52495d64c: Status 404 returned error can't find the container with id 5ac671622017f7e1d511956fff1d1ba9744c20903ac7782f1dd5baa52495d64c
Sep 16 02:18:24.173321 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:24.173303    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5ea2d3_a6ae_4089_8204_d7422983fe38.slice/crio-20bf28afeee57f9c426106b3c7ff49b96cf867a229534c45c775bdef7eac60ae WatchSource:0}: Error finding container 20bf28afeee57f9c426106b3c7ff49b96cf867a229534c45c775bdef7eac60ae: Status 404 returned error can't find the container with id 20bf28afeee57f9c426106b3c7ff49b96cf867a229534c45c775bdef7eac60ae
Sep 16 02:18:24.536936 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:24.536899    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:24.537086 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.537037    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:24.537148 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.537095    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs podName:103d25bf-1d27-4cb9-be5c-cf6a8f8e691b nodeName:}" failed. No retries permitted until 2026-09-16 02:18:26.53708228 +0000 UTC m=+5.967143644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs") pod "network-metrics-daemon-t4vxs" (UID: "103d25bf-1d27-4cb9-be5c-cf6a8f8e691b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:24.637724 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:24.637690    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:24.637901 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.637835    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:24.637901 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.637892    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret podName:9b83e697-837d-4c0e-b454-767d6743f7ab nodeName:}" failed. No retries permitted until 2026-09-16 02:18:26.63787862 +0000 UTC m=+6.067939987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret") pod "global-pull-secret-syncer-plfnq" (UID: "9b83e697-837d-4c0e-b454-767d6743f7ab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:24.738670 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:24.738514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:24.738829 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.738722    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:24.738829 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.738741    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:24.738829 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.738752    2556 projected.go:196] Error preparing data for projected volume kube-api-access-62w8z for pod openshift-network-diagnostics/network-check-target-4sfbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:24.738829 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:24.738819    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z podName:3b94c46a-de62-4b93-ba0c-66059c365abb nodeName:}" failed. No retries permitted until 2026-09-16 02:18:26.738799934 +0000 UTC m=+6.168861317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-62w8z" (UniqueName: "kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z") pod "network-check-target-4sfbv" (UID: "3b94c46a-de62-4b93-ba0c-66059c365abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:24.947772 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:24.947541    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 02:13:22 +0000 UTC" deadline="2028-05-14 18:35:24.17427263 +0000 UTC"
Sep 16 02:18:24.947772 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:24.947609    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14560h16m59.226669284s"
Sep 16 02:18:25.109491 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.109462    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:25.109650 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:25.109582    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:25.110283 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.110251    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:25.110386 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:25.110340    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:25.110461 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.110405    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:25.110533 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:25.110505    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:25.139098 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.138986    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpdvn" event={"ID":"8ea3ca72-53a1-456c-85f1-8750d3ac07b1","Type":"ContainerStarted","Data":"e6b28c5c8b344be291f23039be3e8cbfd36fb27bee1614e7caad93bf2db3ada1"}
Sep 16 02:18:25.143649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.143617    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9w9ds" event={"ID":"6e60ad12-0057-402f-b9fb-56ffa04e13a0","Type":"ContainerStarted","Data":"0ec48270ac6fedd809a8fb75b9fc69ead0239dcd0ae0a2b05f58eeb3779abe83"}
Sep 16 02:18:25.148808 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.148778    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal" event={"ID":"c5e07eb16b748b05b653e47f2fc353a6","Type":"ContainerStarted","Data":"dde6914c60cf7b3879d3aced1b8b10fba9797f2c6b78d762b105e25c54d002b1"}
Sep 16 02:18:25.156765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.156706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj" event={"ID":"e2c8009a-e055-450a-9175-53a0d5291656","Type":"ContainerStarted","Data":"5ac671622017f7e1d511956fff1d1ba9744c20903ac7782f1dd5baa52495d64c"}
Sep 16 02:18:25.167254 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.167221    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5q6nd" event={"ID":"2d5ac133-a00e-48e5-94d6-240cafe853ad","Type":"ContainerStarted","Data":"8da56e3aaaab7322de0b276241ce45bec728f69f56975a10ede3dc4c77770739"}
Sep 16 02:18:25.174917 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.174882    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"e66836c888e2c20c0afe6be89ce2aa35c7a70c21eea9b4c07b9469020abfc959"}
Sep 16 02:18:25.186173 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.185467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rd5kz" event={"ID":"68c43980-26d5-4302-bdc8-127d3fbdffeb","Type":"ContainerStarted","Data":"361204d6b569455994cdaa6e7878d234c3e01305f03c94f882b1d7c93c626b60"}
Sep 16 02:18:25.193570 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.193533    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bzlfw" event={"ID":"7ab38bf5-ae74-4023-8c82-d79ce7db57dd","Type":"ContainerStarted","Data":"36f873ee9785329709da6787f38732a318fce5e1ca842e0cb5d644f973723048"}
Sep 16 02:18:25.206062 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.206022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerStarted","Data":"a1f730555604db953c09ad25908ce20208ecb8c4fd1552caca89bb3b422c6f18"}
Sep 16 02:18:25.227787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.227115    2556 generic.go:356] "Generic (PLEG): container finished" podID="d37433c425db0f461fce4d25fa7def78" containerID="0fd988b548f072bbfee7bbaf364ac352bd945f9077d141e89e064153e483eb4f" exitCode=0
Sep 16 02:18:25.227787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.227408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" event={"ID":"d37433c425db0f461fce4d25fa7def78","Type":"ContainerDied","Data":"0fd988b548f072bbfee7bbaf364ac352bd945f9077d141e89e064153e483eb4f"}
Sep 16 02:18:25.242511 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.241705    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.ec2.internal" podStartSLOduration=3.241684994 podStartE2EDuration="3.241684994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:25.163791983 +0000 UTC m=+4.593853370" watchObservedRunningTime="2026-09-16 02:18:25.241684994 +0000 UTC m=+4.671746383"
Sep 16 02:18:25.243033 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:25.242959    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jn7jj" event={"ID":"5b5ea2d3-a6ae-4089-8204-d7422983fe38","Type":"ContainerStarted","Data":"20bf28afeee57f9c426106b3c7ff49b96cf867a229534c45c775bdef7eac60ae"}
Sep 16 02:18:26.268731 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:26.268650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" event={"ID":"d37433c425db0f461fce4d25fa7def78","Type":"ContainerStarted","Data":"ff06e9a9fd5654cdbff4146434ae899fe72fbcd8bf887f730bca0e0643a37279"}
Sep 16 02:18:26.553474 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:26.553376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:26.553637 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.553559    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:26.553637 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.553632    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs podName:103d25bf-1d27-4cb9-be5c-cf6a8f8e691b nodeName:}" failed. No retries permitted until 2026-09-16 02:18:30.553613859 +0000 UTC m=+9.983675241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs") pod "network-metrics-daemon-t4vxs" (UID: "103d25bf-1d27-4cb9-be5c-cf6a8f8e691b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:26.654311 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:26.654269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:26.654516 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.654443    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:26.654516 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.654512    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret podName:9b83e697-837d-4c0e-b454-767d6743f7ab nodeName:}" failed. No retries permitted until 2026-09-16 02:18:30.654490312 +0000 UTC m=+10.084551698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret") pod "global-pull-secret-syncer-plfnq" (UID: "9b83e697-837d-4c0e-b454-767d6743f7ab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:26.755252 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:26.755212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:26.755470 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.755444    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:26.755470 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.755470    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:26.755593 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.755483    2556 projected.go:196] Error preparing data for projected volume kube-api-access-62w8z for pod openshift-network-diagnostics/network-check-target-4sfbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:26.755593 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:26.755543    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z podName:3b94c46a-de62-4b93-ba0c-66059c365abb nodeName:}" failed. No retries permitted until 2026-09-16 02:18:30.755524228 +0000 UTC m=+10.185585611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-62w8z" (UniqueName: "kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z") pod "network-check-target-4sfbv" (UID: "3b94c46a-de62-4b93-ba0c-66059c365abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:27.108560 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:27.108530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:27.108803 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:27.108536    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:27.108803 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:27.108657    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:27.108803 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:27.108667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:27.108803 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:27.108722    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:27.108803 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:27.108800    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:29.110170 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:29.108082    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:29.110170 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:29.108209    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:29.110170 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:29.108274    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:29.110170 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:29.108395    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:29.110170 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:29.108484    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:29.110170 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:29.108559    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:30.588095 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:30.587989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:30.588622 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.588101    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:30.588622 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.588173    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs podName:103d25bf-1d27-4cb9-be5c-cf6a8f8e691b nodeName:}" failed. No retries permitted until 2026-09-16 02:18:38.588151877 +0000 UTC m=+18.018213255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs") pod "network-metrics-daemon-t4vxs" (UID: "103d25bf-1d27-4cb9-be5c-cf6a8f8e691b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:30.688823 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:30.688785    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:30.689014 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.688923    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:30.689014 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.688983    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret podName:9b83e697-837d-4c0e-b454-767d6743f7ab nodeName:}" failed. No retries permitted until 2026-09-16 02:18:38.688965922 +0000 UTC m=+18.119027289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret") pod "global-pull-secret-syncer-plfnq" (UID: "9b83e697-837d-4c0e-b454-767d6743f7ab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:30.789144 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:30.789107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:30.789314 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.789288    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:30.789397 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.789327    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:30.789397 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.789341    2556 projected.go:196] Error preparing data for projected volume kube-api-access-62w8z for pod openshift-network-diagnostics/network-check-target-4sfbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:30.789517 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:30.789406    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z podName:3b94c46a-de62-4b93-ba0c-66059c365abb nodeName:}" failed. No retries permitted until 2026-09-16 02:18:38.789387111 +0000 UTC m=+18.219448501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-62w8z" (UniqueName: "kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z") pod "network-check-target-4sfbv" (UID: "3b94c46a-de62-4b93-ba0c-66059c365abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:31.108891 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:31.108379    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:31.108891 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:31.108513    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:31.108891 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:31.108865    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:31.109196 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:31.108965    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:31.109196 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:31.108974    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:31.109196 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:31.109090    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:33.107687 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:33.107639    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:33.108158 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:33.107831    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:33.108158 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:33.107882    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:33.108158 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:33.107992    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:33.108158 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:33.108062    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:33.108404 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:33.108180    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:35.108121 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:35.107905    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:35.109031 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:35.107909    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:35.109031 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:35.108227    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:35.109031 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:35.107908    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:35.109031 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:35.108318    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:35.109031 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:35.108366    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:37.107604 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:37.107569    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:37.107604 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:37.107605    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:37.108085 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:37.107569    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:37.108085 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:37.107703    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:37.108085 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:37.107800    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:37.108085 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:37.107875    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:38.647016 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:38.646973    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:38.647551 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.647133    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:38.647551 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.647205    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs podName:103d25bf-1d27-4cb9-be5c-cf6a8f8e691b nodeName:}" failed. No retries permitted until 2026-09-16 02:18:54.647185029 +0000 UTC m=+34.077246393 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs") pod "network-metrics-daemon-t4vxs" (UID: "103d25bf-1d27-4cb9-be5c-cf6a8f8e691b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:38.747973 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:38.747939    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:38.748141 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.748116    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:38.748284 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.748198    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret podName:9b83e697-837d-4c0e-b454-767d6743f7ab nodeName:}" failed. No retries permitted until 2026-09-16 02:18:54.748177325 +0000 UTC m=+34.178238693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret") pod "global-pull-secret-syncer-plfnq" (UID: "9b83e697-837d-4c0e-b454-767d6743f7ab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:38.848331 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:38.848291    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:38.848514 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.848476    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:38.848514 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.848504    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:38.848631 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.848531    2556 projected.go:196] Error preparing data for projected volume kube-api-access-62w8z for pod openshift-network-diagnostics/network-check-target-4sfbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:38.848631 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:38.848599    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z podName:3b94c46a-de62-4b93-ba0c-66059c365abb nodeName:}" failed. No retries permitted until 2026-09-16 02:18:54.848581358 +0000 UTC m=+34.278642738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-62w8z" (UniqueName: "kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z") pod "network-check-target-4sfbv" (UID: "3b94c46a-de62-4b93-ba0c-66059c365abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:39.107869 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:39.107790    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:39.107869 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:39.107832    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:39.108087 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:39.107933    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:39.108087 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:39.107991    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:39.108087 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:39.108060    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:39.108202 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:39.108155    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:41.109648 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:41.109603    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:41.110102 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:41.109767    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:41.110274 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:41.110254    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:41.110404 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:41.110385    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:41.110485 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:41.110473    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:41.110624 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:41.110604    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:42.307020 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.306985    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9w9ds" event={"ID":"6e60ad12-0057-402f-b9fb-56ffa04e13a0","Type":"ContainerStarted","Data":"ad7d7f5b007114042616abc52baa195c6ef73bef5b49628d65225ab82573fb3d"}
Sep 16 02:18:42.308787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.308756    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj" event={"ID":"e2c8009a-e055-450a-9175-53a0d5291656","Type":"ContainerStarted","Data":"c2e550812f4c392bdcb6d482db562f6919b1726e3490e2d122b40ef74b152d2f"}
Sep 16 02:18:42.310401 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.310377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rd5kz" event={"ID":"68c43980-26d5-4302-bdc8-127d3fbdffeb","Type":"ContainerStarted","Data":"172e6712f7883a777d69ad1664266d6d4a7945000ff3019dfbde8068e4341edc"}
Sep 16 02:18:42.312061 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.312040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bzlfw" event={"ID":"7ab38bf5-ae74-4023-8c82-d79ce7db57dd","Type":"ContainerStarted","Data":"476b934f34559a84b1ad81953a5504689cbf6627a24934feb4144d6a12a66439"}
Sep 16 02:18:42.313546 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.313520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerStarted","Data":"a76b5da30ff30abb14aa84af27b2418094c5db55be5d2bb95891684cd3987e49"}
Sep 16 02:18:42.315100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.315077    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jn7jj" event={"ID":"5b5ea2d3-a6ae-4089-8204-d7422983fe38","Type":"ContainerStarted","Data":"9d95b770acf74c756e29e65456a12e055784ec6d99208d30c8a72ad16aed7feb"}
Sep 16 02:18:42.317035 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.317010    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpdvn" event={"ID":"8ea3ca72-53a1-456c-85f1-8750d3ac07b1","Type":"ContainerStarted","Data":"852db2cff340b4d5902d0fb68aa2556175302880fd91fb626b8813b719d3ef25"}
Sep 16 02:18:42.320134 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.320082    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.ec2.internal" podStartSLOduration=20.32006766 podStartE2EDuration="20.32006766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:26.282835005 +0000 UTC m=+5.712896392" watchObservedRunningTime="2026-09-16 02:18:42.32006766 +0000 UTC m=+21.750129048"
Sep 16 02:18:42.332304 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.332244    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9w9ds" podStartSLOduration=3.636943198 podStartE2EDuration="21.332227374s" totalImagesPullingTime="17.695284176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:42.320202012 +0000 UTC m=+21.750263387" watchObservedRunningTime="2026-09-16 02:18:42.332227374 +0000 UTC m=+21.762288759"
Sep 16 02:18:42.332581 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.332560    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jn7jj" podStartSLOduration=3.647255435 podStartE2EDuration="21.332554699s" totalImagesPullingTime="17.685299264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:42.332381407 +0000 UTC m=+21.762442784" watchObservedRunningTime="2026-09-16 02:18:42.332554699 +0000 UTC m=+21.762616085"
Sep 16 02:18:42.351955 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.351894    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rd5kz" podStartSLOduration=3.659826717 podStartE2EDuration="21.351863575s" totalImagesPullingTime="17.692036858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:42.34926137 +0000 UTC m=+21.779322756" watchObservedRunningTime="2026-09-16 02:18:42.351863575 +0000 UTC m=+21.781924959"
Sep 16 02:18:42.392508 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.392111    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tpdvn" podStartSLOduration=3.790211695 podStartE2EDuration="21.392094651s" totalImagesPullingTime="17.601882956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:42.391933156 +0000 UTC m=+21.821994543" watchObservedRunningTime="2026-09-16 02:18:42.392094651 +0000 UTC m=+21.822156036"
Sep 16 02:18:42.406966 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:42.406919    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bzlfw" podStartSLOduration=3.676108953 podStartE2EDuration="21.406902973s" totalImagesPullingTime="17.73079402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:42.406768217 +0000 UTC m=+21.836829622" watchObservedRunningTime="2026-09-16 02:18:42.406902973 +0000 UTC m=+21.836964376"
Sep 16 02:18:43.107188 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.107158    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:43.107355 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.107160    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:43.107355 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.107297    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:43.107355 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:43.107287    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:43.107535 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:43.107449    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:43.107573 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:43.107530    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:43.188060 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.188033    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:43.188838 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.188813    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:43.320638 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.320602    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5q6nd" event={"ID":"2d5ac133-a00e-48e5-94d6-240cafe853ad","Type":"ContainerStarted","Data":"2c85a94811a795a6bac0028869d7cace6b7232e3d065416a19ad5de97746d558"}
Sep 16 02:18:43.323440 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.323400    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"03ff35dc57e72e0115ad35fb80ff78fcfb4b24b664063900d8132e62fbc10889"}
Sep 16 02:18:43.323564 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.323489    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"cc7e1085bd7d6c47e83203bbac012e9d866e9a1a2642fa3007e5a8709807bc94"}
Sep 16 02:18:43.323564 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.323529    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"eb709c0dcdfe8b8aaee7ee5861914cf8baf050a7b4e1a82685b923ec845c64a7"}
Sep 16 02:18:43.323657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.323566    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"90055e9704eaae0240ecd6e695246554cf3c42f194db8eb54750c0a66bfc2f6c"}
Sep 16 02:18:43.323657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.323607    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"aa8a96a9b6f86710982f6b2f1264b8103f28cfcca524b88e865cb94c501b1de5"}
Sep 16 02:18:43.323657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.323636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"e53f480e8734ae331e2f1bdc92b296b2db92382fb57c0138196dc0f3e440d544"}
Sep 16 02:18:43.324793 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.324765    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c014add-556b-46aa-a326-69b9bad2ed52" containerID="a76b5da30ff30abb14aa84af27b2418094c5db55be5d2bb95891684cd3987e49" exitCode=0
Sep 16 02:18:43.324902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.324835    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerDied","Data":"a76b5da30ff30abb14aa84af27b2418094c5db55be5d2bb95891684cd3987e49"}
Sep 16 02:18:43.325576 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.325542    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:43.325874 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.325859    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tpdvn"
Sep 16 02:18:43.335815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.335777    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5q6nd" podStartSLOduration=4.648542255 podStartE2EDuration="22.335763482s" totalImagesPullingTime="17.687221227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:43.33549601 +0000 UTC m=+22.765557395" watchObservedRunningTime="2026-09-16 02:18:43.335763482 +0000 UTC m=+22.765824868"
Sep 16 02:18:43.700063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.700039    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 02:18:43.961659 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.961491    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T02:18:43.700057419Z","UUID":"50f4a4cd-662a-4c7e-ae03-9fb6c9b2bc94","Handler":null,"Name":"","Endpoint":""}
Sep 16 02:18:43.962916 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.962893    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 02:18:43.962916 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:43.962928    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 02:18:44.329962 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:44.329878    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj" event={"ID":"e2c8009a-e055-450a-9175-53a0d5291656","Type":"ContainerStarted","Data":"0525dc1f05b0b9fd4f355bd4dcfe2be566dd0f959fafaed633fe4695d8ccd5e5"}
Sep 16 02:18:45.107071 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:45.107047    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:45.107210 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:45.107047    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:45.107210 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:45.107170    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:45.107332 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:45.107250    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:45.107332 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:45.107058    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:45.107523 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:45.107357    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:45.336185 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:45.336156    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"21218d9d388572938e239f718a9ffd0b245bf8eee606f9bf1ab2fb7403b3607a"}
Sep 16 02:18:46.340308 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:46.340270    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj" event={"ID":"e2c8009a-e055-450a-9175-53a0d5291656","Type":"ContainerStarted","Data":"1115766425c061585a9e401ac4c776abe7ab912e9bab646a995d4c24157ddda1"}
Sep 16 02:18:46.357687 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:46.355874    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cwzj" podStartSLOduration=4.72344788 podStartE2EDuration="25.355854747s" totalImagesPullingTime="20.632406867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:46.354928064 +0000 UTC m=+25.784989449" watchObservedRunningTime="2026-09-16 02:18:46.355854747 +0000 UTC m=+25.785916134"
Sep 16 02:18:47.107511 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:47.107475    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:47.107657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:47.107475    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:47.107657 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:47.107578    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:47.107657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:47.107601    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:47.107787 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:47.107698    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:47.107826 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:47.107793    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:47.347556 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:47.347347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" event={"ID":"725b636c-96b2-4291-9f13-6133cf5ce923","Type":"ContainerStarted","Data":"9efad444d25c3afd8b96d4a81bef7d5ed64d1e6da4d1b6c126ecc4c2a01cf954"}
Sep 16 02:18:47.370216 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:47.370125    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd" podStartSLOduration=8.286261284 podStartE2EDuration="26.370109966s" totalImagesPullingTime="18.083848682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:18:47.369915118 +0000 UTC m=+26.799976508" watchObservedRunningTime="2026-09-16 02:18:47.370109966 +0000 UTC m=+26.800171330"
Sep 16 02:18:48.350367 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:48.350327    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:48.350986 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:48.350416    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:48.350986 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:48.350478    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:48.367826 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:48.367800    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:48.367960 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:48.367907    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:18:49.093491 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:49.093454    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plfnq"]
Sep 16 02:18:49.093665 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:49.093619    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:49.093791 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:49.093766    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:49.097262 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:49.096911    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4vxs"]
Sep 16 02:18:49.097810 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:49.097488    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:49.098315 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:49.097762    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:49.099420 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:49.099391    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4sfbv"]
Sep 16 02:18:49.099550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:49.099507    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:49.099615 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:49.099596    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:50.356613 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:50.356577    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c014add-556b-46aa-a326-69b9bad2ed52" containerID="742e88516b95d12cfebd40067a19a8796f565a908d00d910eb9b07c3b7a4d770" exitCode=0
Sep 16 02:18:50.357245 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:50.356657    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerDied","Data":"742e88516b95d12cfebd40067a19a8796f565a908d00d910eb9b07c3b7a4d770"}
Sep 16 02:18:51.108713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:51.108676    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:51.108902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:51.108763    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:51.108902 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:51.108765    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:51.108902 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:51.108877    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:51.109046 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:51.108906    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:51.109046 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:51.108981    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:52.363933 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:52.363744    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c014add-556b-46aa-a326-69b9bad2ed52" containerID="51b38d08aa3d4db506957b60facc3900e5c149c1dd6e290303e492cf7054c900" exitCode=0
Sep 16 02:18:52.363933 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:52.363826    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerDied","Data":"51b38d08aa3d4db506957b60facc3900e5c149c1dd6e290303e492cf7054c900"}
Sep 16 02:18:53.107368 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:53.107333    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:53.107547 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:53.107332    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:53.107547 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:53.107458    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:53.107547 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:53.107531    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:53.107547 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:53.107339    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:53.107771 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:53.107597    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:54.371938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:54.371907    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c014add-556b-46aa-a326-69b9bad2ed52" containerID="b74abf8448000348e10d27ebad04c6088fe214090069ed836282d6f29bc50966" exitCode=0
Sep 16 02:18:54.372316 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:54.371951    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerDied","Data":"b74abf8448000348e10d27ebad04c6088fe214090069ed836282d6f29bc50966"}
Sep 16 02:18:54.660531 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:54.660498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:54.660773 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.660630    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:54.660773 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.660697    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs podName:103d25bf-1d27-4cb9-be5c-cf6a8f8e691b nodeName:}" failed. No retries permitted until 2026-09-16 02:19:26.6606777 +0000 UTC m=+66.090739080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs") pod "network-metrics-daemon-t4vxs" (UID: "103d25bf-1d27-4cb9-be5c-cf6a8f8e691b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 02:18:54.760794 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:54.760761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:54.760972 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.760862    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:54.760972 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.760908    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret podName:9b83e697-837d-4c0e-b454-767d6743f7ab nodeName:}" failed. No retries permitted until 2026-09-16 02:19:26.760895911 +0000 UTC m=+66.190957275 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret") pod "global-pull-secret-syncer-plfnq" (UID: "9b83e697-837d-4c0e-b454-767d6743f7ab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 02:18:54.861580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:54.861550    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:54.861764 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.861682    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 02:18:54.861764 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.861695    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 02:18:54.861764 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.861704    2556 projected.go:196] Error preparing data for projected volume kube-api-access-62w8z for pod openshift-network-diagnostics/network-check-target-4sfbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:54.861764 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:54.861747    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z podName:3b94c46a-de62-4b93-ba0c-66059c365abb nodeName:}" failed. No retries permitted until 2026-09-16 02:19:26.861734653 +0000 UTC m=+66.291796018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-62w8z" (UniqueName: "kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z") pod "network-check-target-4sfbv" (UID: "3b94c46a-de62-4b93-ba0c-66059c365abb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 02:18:55.107863 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.107777    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:55.108103 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:55.107910    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfnq" podUID="9b83e697-837d-4c0e-b454-767d6743f7ab"
Sep 16 02:18:55.108103 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.107777    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:55.108103 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:55.108036    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4sfbv" podUID="3b94c46a-de62-4b93-ba0c-66059c365abb"
Sep 16 02:18:55.108103 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.108068    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:55.108369 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:55.108152    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4vxs" podUID="103d25bf-1d27-4cb9-be5c-cf6a8f8e691b"
Sep 16 02:18:55.874463 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.874374    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-48.ec2.internal" event="NodeReady"
Sep 16 02:18:55.874848 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.874548    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 02:18:55.908364 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.908331    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6795b44c76-kkdw4"]
Sep 16 02:18:55.939818 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.939788    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dcckb"]
Sep 16 02:18:55.939982 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.939910    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:55.942234 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.942209    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 02:18:55.942376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.942209    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 02:18:55.942747 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.942636    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4mjd\""
Sep 16 02:18:55.942747 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.942703    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 02:18:55.948025 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.947994    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 02:18:55.954156 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.954126    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6795b44c76-kkdw4"]
Sep 16 02:18:55.954263 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.954173    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gpkxq"]
Sep 16 02:18:55.954263 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.954215    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:55.956564 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.956537    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 02:18:55.957149 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.957125    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 02:18:55.957250 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.957147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w4x8g\""
Sep 16 02:18:55.970676 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.970608    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gpkxq"]
Sep 16 02:18:55.970676 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.970647    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dcckb"]
Sep 16 02:18:55.970885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.970747    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:55.973411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.973338    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 02:18:55.973411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.973379    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 02:18:55.973411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.973380    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p58tv\""
Sep 16 02:18:55.973653 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:55.973414    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 02:18:56.070747 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070710    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.070747 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070747    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-certificates\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.070958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlb4l\" (UniqueName: \"kubernetes.io/projected/7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6-kube-api-access-rlb4l\") pod \"ingress-canary-gpkxq\" (UID: \"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6\") " pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:56.070958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dqv\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-kube-api-access-89dqv\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.070958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070869    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-installation-pull-secrets\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.070958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-trusted-ca\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.070958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070912    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.071197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.070964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe7d06d-b82f-407c-9b61-73189c034507-tmp-dir\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.071197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.071005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-bound-sa-token\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.071197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.071054    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-image-registry-private-configuration\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.071197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.071083    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6-cert\") pod \"ingress-canary-gpkxq\" (UID: \"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6\") " pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:56.071197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.071120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe7d06d-b82f-407c-9b61-73189c034507-config-volume\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.071197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.071162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wbf\" (UniqueName: \"kubernetes.io/projected/7fe7d06d-b82f-407c-9b61-73189c034507-kube-api-access-v4wbf\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.071197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.071188    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c58df75d-dfcd-45eb-8778-e57ae06efc8d-ca-trust-extracted\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.171827 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171741    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wbf\" (UniqueName: \"kubernetes.io/projected/7fe7d06d-b82f-407c-9b61-73189c034507-kube-api-access-v4wbf\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.171827 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171783    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c58df75d-dfcd-45eb-8778-e57ae06efc8d-ca-trust-extracted\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171830    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171851    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-certificates\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlb4l\" (UniqueName: \"kubernetes.io/projected/7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6-kube-api-access-rlb4l\") pod \"ingress-canary-gpkxq\" (UID: \"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6\") " pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171907    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dqv\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-kube-api-access-89dqv\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-installation-pull-secrets\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-trusted-ca\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.171978    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.172002    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6795b44c76-kkdw4: secret "image-registry-tls" not found
Sep 16 02:18:56.172048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.171988    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.172061    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls podName:c58df75d-dfcd-45eb-8778-e57ae06efc8d nodeName:}" failed. No retries permitted until 2026-09-16 02:18:56.672042965 +0000 UTC m=+36.102104329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls") pod "image-registry-6795b44c76-kkdw4" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d") : secret "image-registry-tls" not found
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.172073    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172095    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe7d06d-b82f-407c-9b61-73189c034507-tmp-dir\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.172122    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls podName:7fe7d06d-b82f-407c-9b61-73189c034507 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:56.672106717 +0000 UTC m=+36.102168096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls") pod "dns-default-dcckb" (UID: "7fe7d06d-b82f-407c-9b61-73189c034507") : secret "dns-default-metrics-tls" not found
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-bound-sa-token\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172198    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-image-registry-private-configuration\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172223    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6-cert\") pod \"ingress-canary-gpkxq\" (UID: \"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6\") " pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172264    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe7d06d-b82f-407c-9b61-73189c034507-config-volume\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c58df75d-dfcd-45eb-8778-e57ae06efc8d-ca-trust-extracted\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe7d06d-b82f-407c-9b61-73189c034507-tmp-dir\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.172967 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172546    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-certificates\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.172967 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.172855    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe7d06d-b82f-407c-9b61-73189c034507-config-volume\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.173397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.173355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-trusted-ca\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.176614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.176590    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-installation-pull-secrets\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.176614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.176601    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-image-registry-private-configuration\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.176767 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.176626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6-cert\") pod \"ingress-canary-gpkxq\" (UID: \"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6\") " pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:56.180343 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.180321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dqv\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-kube-api-access-89dqv\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.180449 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.180352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlb4l\" (UniqueName: \"kubernetes.io/projected/7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6-kube-api-access-rlb4l\") pod \"ingress-canary-gpkxq\" (UID: \"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6\") " pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:56.180597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.180576    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-bound-sa-token\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.181160 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.181123    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wbf\" (UniqueName: \"kubernetes.io/projected/7fe7d06d-b82f-407c-9b61-73189c034507-kube-api-access-v4wbf\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.280448 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.280394    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gpkxq"
Sep 16 02:18:56.619256 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.619180    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gpkxq"]
Sep 16 02:18:56.622841 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:56.622809    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7815c4f9_893e_44ba_a7dd_3fcb75aa0ab6.slice/crio-d0b30ebbfe2f032cff6a542f606ad5c8826de3f3f65637e14cd0b07d1e20697a WatchSource:0}: Error finding container d0b30ebbfe2f032cff6a542f606ad5c8826de3f3f65637e14cd0b07d1e20697a: Status 404 returned error can't find the container with id d0b30ebbfe2f032cff6a542f606ad5c8826de3f3f65637e14cd0b07d1e20697a
Sep 16 02:18:56.677447 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.677399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:56.677627 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.677480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:56.677627 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.677556    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 02:18:56.677627 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.677580    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6795b44c76-kkdw4: secret "image-registry-tls" not found
Sep 16 02:18:56.677627 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.677610    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:18:56.677841 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.677671    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls podName:c58df75d-dfcd-45eb-8778-e57ae06efc8d nodeName:}" failed. No retries permitted until 2026-09-16 02:18:57.677651093 +0000 UTC m=+37.107712460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls") pod "image-registry-6795b44c76-kkdw4" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d") : secret "image-registry-tls" not found
Sep 16 02:18:56.677841 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.677691    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls podName:7fe7d06d-b82f-407c-9b61-73189c034507 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:57.677681685 +0000 UTC m=+37.107743052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls") pod "dns-default-dcckb" (UID: "7fe7d06d-b82f-407c-9b61-73189c034507") : secret "dns-default-metrics-tls" not found
Sep 16 02:18:56.710868 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.710832    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"]
Sep 16 02:18:56.741031 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.740998    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"]
Sep 16 02:18:56.741192 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.741125    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"
Sep 16 02:18:56.743494 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.743461    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.743687 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.743565    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hf8hn\""
Sep 16 02:18:56.743687 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.743597    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.754736 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.754714    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"]
Sep 16 02:18:56.754845 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.754833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"
Sep 16 02:18:56.757233 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.756995    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.757233 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.757066    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.757233 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.757224    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8r2wn\""
Sep 16 02:18:56.772976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.772952    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"]
Sep 16 02:18:56.773080 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.773065    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.775305 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.775278    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 02:18:56.775404 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.775305    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2w29n\""
Sep 16 02:18:56.775404 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.775343    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 02:18:56.775404 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.775348    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.775404 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.775355    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.795012 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.794985    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"]
Sep 16 02:18:56.795012 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.795017    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"]
Sep 16 02:18:56.795167 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.795031    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"]
Sep 16 02:18:56.795167 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.795103    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.795167 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.795113    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"]
Sep 16 02:18:56.797407 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.797387    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.797574 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.797554    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dpl6r\""
Sep 16 02:18:56.797675 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.797653    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 02:18:56.798177 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.798161    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"]
Sep 16 02:18:56.799652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.799633    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.812264 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.811924    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 02:18:56.813545 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.813328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:56.819074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.815989    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 02:18:56.819074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.816520    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-hs4wt"]
Sep 16 02:18:56.819074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.817055    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.819074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.817387    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wj45q\""
Sep 16 02:18:56.819074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.817691    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 02:18:56.819074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.818787    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.841194 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.841170    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"]
Sep 16 02:18:56.841333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.841315    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:56.843728 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.843506    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 02:18:56.843728 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.843540    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 02:18:56.843905 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.843761    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.843905 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.843784    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4cjkj\""
Sep 16 02:18:56.844664 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.844635    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.848257 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.848231    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 02:18:56.863354 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.863319    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-9bc99d698-k2qmx"]
Sep 16 02:18:56.863502 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.863473    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:56.865826 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.865602    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tnzdb\""
Sep 16 02:18:56.865826 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.865652    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.865826 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.865719    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 02:18:56.865826 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.865739    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.878642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878619    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.879246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.879246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878698    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thfls\" (UniqueName: \"kubernetes.io/projected/4d7b4f33-5eee-47b6-ab81-229546cf3383-kube-api-access-thfls\") pod \"network-check-source-6f8b768455-4d79x\" (UID: \"4d7b4f33-5eee-47b6-ab81-229546cf3383\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"
Sep 16 02:18:56.879246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj8gc\" (UniqueName: \"kubernetes.io/projected/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-kube-api-access-mj8gc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.879246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878823    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.879246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btzrq\" (UniqueName: \"kubernetes.io/projected/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-kube-api-access-btzrq\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.879246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5nz\" (UniqueName: \"kubernetes.io/projected/d29dc62f-9e58-48a5-b87f-074fe8f27b44-kube-api-access-2d5nz\") pod \"volume-data-source-validator-86695b7cc7-8qh6p\" (UID: \"d29dc62f-9e58-48a5-b87f-074fe8f27b44\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"
Sep 16 02:18:56.879246 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.878949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.887584 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.887566    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"]
Sep 16 02:18:56.887690 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.887600    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"]
Sep 16 02:18:56.887690 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.887629    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hs4wt"]
Sep 16 02:18:56.887690 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.887651    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9bc99d698-k2qmx"]
Sep 16 02:18:56.887799 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.887690    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:56.890364 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.890343    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 02:18:56.890484 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.890392    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 02:18:56.890553 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.890483    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 02:18:56.890553 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.890352    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8rl2z\""
Sep 16 02:18:56.890655 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.890609    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.890706 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.890690    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 02:18:56.890760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.890722    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.904317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.904290    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"]
Sep 16 02:18:56.917470 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.917444    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mgds9"]
Sep 16 02:18:56.917587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.917570    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:56.919662 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.919640    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 02:18:56.919750 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.919721    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 02:18:56.919750 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.919734    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8kmhr\""
Sep 16 02:18:56.927578 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.927559    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"]
Sep 16 02:18:56.927661 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.927585    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mgds9"]
Sep 16 02:18:56.927661 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.927651    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:56.929787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.929766    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 02:18:56.929870 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.929830    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 02:18:56.929935 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.929895    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d8t2f\""
Sep 16 02:18:56.929935 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.929910    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 02:18:56.930034 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.929965    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 02:18:56.939991 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.938795    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 02:18:56.968066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.968033    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tpdvn_8ea3ca72-53a1-456c-85f1-8750d3ac07b1/konnectivity-agent/0.log" limit=1024
Sep 16 02:18:56.979965 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.979938    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:56.980043 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.979986    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-config\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:56.980043 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thfls\" (UniqueName: \"kubernetes.io/projected/4d7b4f33-5eee-47b6-ab81-229546cf3383-kube-api-access-thfls\") pod \"network-check-source-6f8b768455-4d79x\" (UID: \"4d7b4f33-5eee-47b6-ab81-229546cf3383\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"
Sep 16 02:18:56.980105 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj8gc\" (UniqueName: \"kubernetes.io/projected/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-kube-api-access-mj8gc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.980139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980106    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.980139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:56.980204 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980150    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:56.980204 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980167    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:56.980204 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-tmp\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:56.980322 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980213    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.980322 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.980391 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-default-certificate\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:56.980391 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980366    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-serving-cert\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:56.980391 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.980372    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 02:18:56.980528 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:56.980443    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls podName:8fb91d7c-c3a9-40f3-89fa-8aac191a07ef nodeName:}" failed. No retries permitted until 2026-09-16 02:18:57.480409665 +0000 UTC m=+36.910471051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4skl4" (UID: "8fb91d7c-c3a9-40f3-89fa-8aac191a07ef") : secret "cluster-monitoring-operator-tls" not found
Sep 16 02:18:56.980528 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7m2\" (UniqueName: \"kubernetes.io/projected/d501b79f-f7db-41af-93d6-ae81d067cb5d-kube-api-access-dk7m2\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:56.980618 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980551    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklsz\" (UniqueName: \"kubernetes.io/projected/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-kube-api-access-fklsz\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:56.980618 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980578    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-stats-auth\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:56.980618 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-service-ca-bundle\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:56.980741 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980632    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-serving-cert\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:56.980741 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btzrq\" (UniqueName: \"kubernetes.io/projected/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-kube-api-access-btzrq\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.980741 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980680    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5nz\" (UniqueName: \"kubernetes.io/projected/d29dc62f-9e58-48a5-b87f-074fe8f27b44-kube-api-access-2d5nz\") pod \"volume-data-source-validator-86695b7cc7-8qh6p\" (UID: \"d29dc62f-9e58-48a5-b87f-074fe8f27b44\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"
Sep 16 02:18:56.980741 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980698    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4p7d\" (UniqueName: \"kubernetes.io/projected/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-kube-api-access-q4p7d\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:56.980741 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980714    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2smvw\" (UniqueName: \"kubernetes.io/projected/27b8d016-9177-472c-bb29-5a71c0a14784-kube-api-access-2smvw\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:56.980741 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980737    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.981014 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980767    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-snapshots\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:56.981014 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.980827    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.981287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.981271    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.982882 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.982865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.993026 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.992998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thfls\" (UniqueName: \"kubernetes.io/projected/4d7b4f33-5eee-47b6-ab81-229546cf3383-kube-api-access-thfls\") pod \"network-check-source-6f8b768455-4d79x\" (UID: \"4d7b4f33-5eee-47b6-ab81-229546cf3383\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"
Sep 16 02:18:56.993163 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.993035    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj8gc\" (UniqueName: \"kubernetes.io/projected/dd5bdaad-9ef0-499f-a136-fefde5f6f7d2-kube-api-access-mj8gc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-5gnj8\" (UID: \"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:56.993229 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.993184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btzrq\" (UniqueName: \"kubernetes.io/projected/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-kube-api-access-btzrq\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:56.993501 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:56.993479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5nz\" (UniqueName: \"kubernetes.io/projected/d29dc62f-9e58-48a5-b87f-074fe8f27b44-kube-api-access-2d5nz\") pod \"volume-data-source-validator-86695b7cc7-8qh6p\" (UID: \"d29dc62f-9e58-48a5-b87f-074fe8f27b44\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"
Sep 16 02:18:57.047886 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.047851    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tpdvn_8ea3ca72-53a1-456c-85f1-8750d3ac07b1/konnectivity-agent/0.log" limit=1024
Sep 16 02:18:57.051551 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.051505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"
Sep 16 02:18:57.065102 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.065079    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"
Sep 16 02:18:57.081225 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/512c9eff-f1e7-4507-9d75-fcaf212112e4-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:57.081379 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081244    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.081379 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081316    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-config\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:57.081379 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081355    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:57.081379 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.081371    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 02:18:57.081562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsn95\" (UniqueName: \"kubernetes.io/projected/97829f87-c6a8-4a47-a358-df8b7eb261dd-kube-api-access-dsn95\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.081562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97829f87-c6a8-4a47-a358-df8b7eb261dd-config\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.081562 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.081461    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:18:57.581440382 +0000 UTC m=+37.011501760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : secret "router-metrics-certs-default" not found
Sep 16 02:18:57.081562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081482    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97829f87-c6a8-4a47-a358-df8b7eb261dd-serving-cert\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.081562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97829f87-c6a8-4a47-a358-df8b7eb261dd-trusted-ca\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.081814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.081814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081617    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.081814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:57.081814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081706    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-tmp\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.081814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081744    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-default-certificate\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.081814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-serving-cert\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081838    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7m2\" (UniqueName: \"kubernetes.io/projected/d501b79f-f7db-41af-93d6-ae81d067cb5d-kube-api-access-dk7m2\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.081860    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:18:57.581842833 +0000 UTC m=+37.011904220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : configmap references non-existent config key: service-ca.crt
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.081882    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081924    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklsz\" (UniqueName: \"kubernetes.io/projected/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-kube-api-access-fklsz\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.081949    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls podName:27b8d016-9177-472c-bb29-5a71c0a14784 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:57.581933343 +0000 UTC m=+37.011994717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-t8w94" (UID: "27b8d016-9177-472c-bb29-5a71c0a14784") : secret "samples-operator-tls" not found
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.081981    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-stats-auth\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-service-ca-bundle\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-config\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-serving-cert\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082074    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4p7d\" (UniqueName: \"kubernetes.io/projected/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-kube-api-access-q4p7d\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.082100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2smvw\" (UniqueName: \"kubernetes.io/projected/27b8d016-9177-472c-bb29-5a71c0a14784-kube-api-access-2smvw\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:57.082597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082133    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-snapshots\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.082597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-tmp\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.082597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082543    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.082823 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-snapshots\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.082878 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082843    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"
Sep 16 02:18:57.083015 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.082993    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-service-ca-bundle\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.085233 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.085191    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-serving-cert\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:57.086043 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.085725    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-serving-cert\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.086609 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.086212    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-default-certificate\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.086609 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.086571    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-stats-auth\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.089531 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.089512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7m2\" (UniqueName: \"kubernetes.io/projected/d501b79f-f7db-41af-93d6-ae81d067cb5d-kube-api-access-dk7m2\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.091081 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.091041    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2smvw\" (UniqueName: \"kubernetes.io/projected/27b8d016-9177-472c-bb29-5a71c0a14784-kube-api-access-2smvw\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:57.091392 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.091358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4p7d\" (UniqueName: \"kubernetes.io/projected/177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42-kube-api-access-q4p7d\") pod \"insights-operator-5844b856db-hs4wt\" (UID: \"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42\") " pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.092248 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.092209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklsz\" (UniqueName: \"kubernetes.io/projected/36e5df43-16b6-4495-8396-ac2cc0c0bd0a-kube-api-access-fklsz\") pod \"service-ca-operator-cf7c776b7-kkwcl\" (UID: \"36e5df43-16b6-4495-8396-ac2cc0c0bd0a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:57.109475 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.108186    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:18:57.109475 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.108970    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:18:57.110207 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.109697    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:18:57.111840 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.110824    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7ktl5\""
Sep 16 02:18:57.111840 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.111490    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss74q\""
Sep 16 02:18:57.111840 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.111707    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 02:18:57.112065 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.111846    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 02:18:57.131315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.131228    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"
Sep 16 02:18:57.144767 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.144712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tpdvn_8ea3ca72-53a1-456c-85f1-8750d3ac07b1/konnectivity-agent/0.log" limit=1024
Sep 16 02:18:57.152628 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.152606    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hs4wt"
Sep 16 02:18:57.184120 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.183451    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/512c9eff-f1e7-4507-9d75-fcaf212112e4-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.184379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.184419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsn95\" (UniqueName: \"kubernetes.io/projected/97829f87-c6a8-4a47-a358-df8b7eb261dd-kube-api-access-dsn95\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.184470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97829f87-c6a8-4a47-a358-df8b7eb261dd-config\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.184492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97829f87-c6a8-4a47-a358-df8b7eb261dd-serving-cert\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.184551    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97829f87-c6a8-4a47-a358-df8b7eb261dd-trusted-ca\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.185471    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97829f87-c6a8-4a47-a358-df8b7eb261dd-trusted-ca\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.186067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/512c9eff-f1e7-4507-9d75-fcaf212112e4-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.186162    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.186219    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert podName:512c9eff-f1e7-4507-9d75-fcaf212112e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:57.686200468 +0000 UTC m=+37.116261848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rnf47" (UID: "512c9eff-f1e7-4507-9d75-fcaf212112e4") : secret "networking-console-plugin-cert" not found
Sep 16 02:18:57.187251 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.187228    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97829f87-c6a8-4a47-a358-df8b7eb261dd-config\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.190082 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.190040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97829f87-c6a8-4a47-a358-df8b7eb261dd-serving-cert\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.208779 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.208713    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsn95\" (UniqueName: \"kubernetes.io/projected/97829f87-c6a8-4a47-a358-df8b7eb261dd-kube-api-access-dsn95\") pod \"console-operator-696884fb9c-mgds9\" (UID: \"97829f87-c6a8-4a47-a358-df8b7eb261dd\") " pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.238034 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.237747    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:18:57.336825 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.336508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p"]
Sep 16 02:18:57.340749 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:57.340715    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29dc62f_9e58_48a5_b87f_074fe8f27b44.slice/crio-4ea0c40e5db4a74a8e09c5af2845bb1c08476563237a9e95fb8ac77d4f992763 WatchSource:0}: Error finding container 4ea0c40e5db4a74a8e09c5af2845bb1c08476563237a9e95fb8ac77d4f992763: Status 404 returned error can't find the container with id 4ea0c40e5db4a74a8e09c5af2845bb1c08476563237a9e95fb8ac77d4f992763
Sep 16 02:18:57.362728 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.362652    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-4d79x"]
Sep 16 02:18:57.365606 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:57.365572    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7b4f33_5eee_47b6_ab81_229546cf3383.slice/crio-ffbee9bd026fa555879bd0a897b84b3280ab9d33342c7e4c317425922af534ae WatchSource:0}: Error finding container ffbee9bd026fa555879bd0a897b84b3280ab9d33342c7e4c317425922af534ae: Status 404 returned error can't find the container with id ffbee9bd026fa555879bd0a897b84b3280ab9d33342c7e4c317425922af534ae
Sep 16 02:18:57.381529 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.381363    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x" event={"ID":"4d7b4f33-5eee-47b6-ab81-229546cf3383","Type":"ContainerStarted","Data":"ffbee9bd026fa555879bd0a897b84b3280ab9d33342c7e4c317425922af534ae"}
Sep 16 02:18:57.382915 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.382887    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p" event={"ID":"d29dc62f-9e58-48a5-b87f-074fe8f27b44","Type":"ContainerStarted","Data":"4ea0c40e5db4a74a8e09c5af2845bb1c08476563237a9e95fb8ac77d4f992763"}
Sep 16 02:18:57.384489 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.384397    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gpkxq" event={"ID":"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6","Type":"ContainerStarted","Data":"d0b30ebbfe2f032cff6a542f606ad5c8826de3f3f65637e14cd0b07d1e20697a"}
Sep 16 02:18:57.391263 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.391240    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8"]
Sep 16 02:18:57.395244 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:57.395216    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5bdaad_9ef0_499f_a136_fefde5f6f7d2.slice/crio-0bf56766bcafcbed32f5be02d19cce747fa15fca643490049f50c87cd1289c1a WatchSource:0}: Error finding container 0bf56766bcafcbed32f5be02d19cce747fa15fca643490049f50c87cd1289c1a: Status 404 returned error can't find the container with id 0bf56766bcafcbed32f5be02d19cce747fa15fca643490049f50c87cd1289c1a
Sep 16 02:18:57.433871 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.433799    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl"]
Sep 16 02:18:57.437588 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:57.437559    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e5df43_16b6_4495_8396_ac2cc0c0bd0a.slice/crio-d083451ec74cb59397396460e55724e57f92637decb9036658e40998f59842a0 WatchSource:0}: Error finding container d083451ec74cb59397396460e55724e57f92637decb9036658e40998f59842a0: Status 404 returned error can't find the container with id d083451ec74cb59397396460e55724e57f92637decb9036658e40998f59842a0
Sep 16 02:18:57.452186 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.452165    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hs4wt"]
Sep 16 02:18:57.454711 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:57.454658    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177bd2fb_c14b_4d9e_aae0_a92b2dfe9f42.slice/crio-1af9bd4254f1909b3e96f306964bb40d0e0fd55e3d3eec122acf209b1c9a8fb6 WatchSource:0}: Error finding container 1af9bd4254f1909b3e96f306964bb40d0e0fd55e3d3eec122acf209b1c9a8fb6: Status 404 returned error can't find the container with id 1af9bd4254f1909b3e96f306964bb40d0e0fd55e3d3eec122acf209b1c9a8fb6
Sep 16 02:18:57.487490 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.487453    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:57.487646 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.487612    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 02:18:57.487697 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.487689    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls podName:8fb91d7c-c3a9-40f3-89fa-8aac191a07ef nodeName:}" failed. No retries permitted until 2026-09-16 02:18:58.487670221 +0000 UTC m=+37.917731603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4skl4" (UID: "8fb91d7c-c3a9-40f3-89fa-8aac191a07ef") : secret "cluster-monitoring-operator-tls" not found
Sep 16 02:18:57.524411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.524330    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mgds9"]
Sep 16 02:18:57.527944 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:18:57.527908    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97829f87_c6a8_4a47_a358_df8b7eb261dd.slice/crio-b8997ceea22207e85ca01c47935ff1bf60b13a2bff5136dc4bb2b62ad9345e4e WatchSource:0}: Error finding container b8997ceea22207e85ca01c47935ff1bf60b13a2bff5136dc4bb2b62ad9345e4e: Status 404 returned error can't find the container with id b8997ceea22207e85ca01c47935ff1bf60b13a2bff5136dc4bb2b62ad9345e4e
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.588758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.588868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.588900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.589085    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.589148    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls podName:27b8d016-9177-472c-bb29-5a71c0a14784 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:58.589129898 +0000 UTC m=+38.019191283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-t8w94" (UID: "27b8d016-9177-472c-bb29-5a71c0a14784") : secret "samples-operator-tls" not found
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.589192    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.589238    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:18:58.589226387 +0000 UTC m=+38.019287768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : secret "router-metrics-certs-default" not found
Sep 16 02:18:57.589288 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.589290    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:18:58.589271405 +0000 UTC m=+38.019332775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : configmap references non-existent config key: service-ca.crt
Sep 16 02:18:57.689515 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.689470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:57.689712 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.689535    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:57.689712 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:57.689632    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:57.689831 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.689810    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 02:18:57.689897 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.689879    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert podName:512c9eff-f1e7-4507-9d75-fcaf212112e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:58.689859195 +0000 UTC m=+38.119920579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rnf47" (UID: "512c9eff-f1e7-4507-9d75-fcaf212112e4") : secret "networking-console-plugin-cert" not found
Sep 16 02:18:57.690035 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.690014    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:18:57.690085 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.690062    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 02:18:57.690085 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.690082    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6795b44c76-kkdw4: secret "image-registry-tls" not found
Sep 16 02:18:57.690199 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.690086    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls podName:7fe7d06d-b82f-407c-9b61-73189c034507 nodeName:}" failed. No retries permitted until 2026-09-16 02:18:59.690060346 +0000 UTC m=+39.120121726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls") pod "dns-default-dcckb" (UID: "7fe7d06d-b82f-407c-9b61-73189c034507") : secret "dns-default-metrics-tls" not found
Sep 16 02:18:57.690199 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:57.690130    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls podName:c58df75d-dfcd-45eb-8778-e57ae06efc8d nodeName:}" failed. No retries permitted until 2026-09-16 02:18:59.690113243 +0000 UTC m=+39.120174610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls") pod "image-registry-6795b44c76-kkdw4" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d") : secret "image-registry-tls" not found
Sep 16 02:18:58.404193 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.403967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl" event={"ID":"36e5df43-16b6-4495-8396-ac2cc0c0bd0a","Type":"ContainerStarted","Data":"d083451ec74cb59397396460e55724e57f92637decb9036658e40998f59842a0"}
Sep 16 02:18:58.407267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.407114    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hs4wt" event={"ID":"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42","Type":"ContainerStarted","Data":"1af9bd4254f1909b3e96f306964bb40d0e0fd55e3d3eec122acf209b1c9a8fb6"}
Sep 16 02:18:58.413315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.413252    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" event={"ID":"97829f87-c6a8-4a47-a358-df8b7eb261dd","Type":"ContainerStarted","Data":"b8997ceea22207e85ca01c47935ff1bf60b13a2bff5136dc4bb2b62ad9345e4e"}
Sep 16 02:18:58.418499 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.418470    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8" event={"ID":"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2","Type":"ContainerStarted","Data":"0bf56766bcafcbed32f5be02d19cce747fa15fca643490049f50c87cd1289c1a"}
Sep 16 02:18:58.498621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.497866    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:18:58.498621 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.498182    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 02:18:58.498621 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.498249    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls podName:8fb91d7c-c3a9-40f3-89fa-8aac191a07ef nodeName:}" failed. No retries permitted until 2026-09-16 02:19:00.498229473 +0000 UTC m=+39.928290842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4skl4" (UID: "8fb91d7c-c3a9-40f3-89fa-8aac191a07ef") : secret "cluster-monitoring-operator-tls" not found
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.599209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.599322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.599352    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.599568    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.599632    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls podName:27b8d016-9177-472c-bb29-5a71c0a14784 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:00.599613216 +0000 UTC m=+40.029674583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-t8w94" (UID: "27b8d016-9177-472c-bb29-5a71c0a14784") : secret "samples-operator-tls" not found
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.600021    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.600064    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:00.60004993 +0000 UTC m=+40.030111336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : secret "router-metrics-certs-default" not found
Sep 16 02:18:58.600165 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.600130    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:00.600119786 +0000 UTC m=+40.030181154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : configmap references non-existent config key: service-ca.crt
Sep 16 02:18:58.700760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:58.700149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:18:58.700760 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.700315    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 02:18:58.700760 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:58.700379    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert podName:512c9eff-f1e7-4507-9d75-fcaf212112e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:00.700361435 +0000 UTC m=+40.130422817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rnf47" (UID: "512c9eff-f1e7-4507-9d75-fcaf212112e4") : secret "networking-console-plugin-cert" not found
Sep 16 02:18:59.711474 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:59.711190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:18:59.711474 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:18:59.711265    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:18:59.711474 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:59.711468    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 02:18:59.711474 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:59.711486    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6795b44c76-kkdw4: secret "image-registry-tls" not found
Sep 16 02:18:59.712177 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:59.711548    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls podName:c58df75d-dfcd-45eb-8778-e57ae06efc8d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:03.711529518 +0000 UTC m=+43.141590888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls") pod "image-registry-6795b44c76-kkdw4" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d") : secret "image-registry-tls" not found
Sep 16 02:18:59.712177 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:59.712161    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:18:59.712300 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:18:59.712209    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls podName:7fe7d06d-b82f-407c-9b61-73189c034507 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:03.712194799 +0000 UTC m=+43.142256169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls") pod "dns-default-dcckb" (UID: "7fe7d06d-b82f-407c-9b61-73189c034507") : secret "dns-default-metrics-tls" not found
Sep 16 02:19:00.404371 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:00.404319    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tpdvn_8ea3ca72-53a1-456c-85f1-8750d3ac07b1/konnectivity-agent/0.log" limit=1024
Sep 16 02:19:00.461738 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:00.461699    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tpdvn_8ea3ca72-53a1-456c-85f1-8750d3ac07b1/konnectivity-agent/0.log" limit=1024
Sep 16 02:19:00.519369 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:00.519324    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:19:00.519556 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.519466    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 02:19:00.519638 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.519554    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls podName:8fb91d7c-c3a9-40f3-89fa-8aac191a07ef nodeName:}" failed. No retries permitted until 2026-09-16 02:19:04.519530216 +0000 UTC m=+43.949591664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4skl4" (UID: "8fb91d7c-c3a9-40f3-89fa-8aac191a07ef") : secret "cluster-monitoring-operator-tls" not found
Sep 16 02:19:00.619982 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:00.619946    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:00.620166 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:00.619997    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:19:00.620166 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:00.620109    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:00.620166 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.620144    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:04.620119912 +0000 UTC m=+44.050181297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : configmap references non-existent config key: service-ca.crt
Sep 16 02:19:00.620347 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.620184    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 02:19:00.620347 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.620256    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls podName:27b8d016-9177-472c-bb29-5a71c0a14784 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:04.620230625 +0000 UTC m=+44.050292013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-t8w94" (UID: "27b8d016-9177-472c-bb29-5a71c0a14784") : secret "samples-operator-tls" not found
Sep 16 02:19:00.620347 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.620194    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 02:19:00.620347 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.620335    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:04.620320265 +0000 UTC m=+44.050381646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : secret "router-metrics-certs-default" not found
Sep 16 02:19:00.721712 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:00.721672    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:19:00.722178 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.721866    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 02:19:00.722178 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:00.721951    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert podName:512c9eff-f1e7-4507-9d75-fcaf212112e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:04.721929465 +0000 UTC m=+44.151990832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rnf47" (UID: "512c9eff-f1e7-4507-9d75-fcaf212112e4") : secret "networking-console-plugin-cert" not found
Sep 16 02:19:03.751830 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:03.751782    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:19:03.752224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:03.751854    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:19:03.752224 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:03.751959    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 02:19:03.752224 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:03.751985    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6795b44c76-kkdw4: secret "image-registry-tls" not found
Sep 16 02:19:03.752224 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:03.752006    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:19:03.752224 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:03.752055    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls podName:c58df75d-dfcd-45eb-8778-e57ae06efc8d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:11.752032863 +0000 UTC m=+51.182094246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls") pod "image-registry-6795b44c76-kkdw4" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d") : secret "image-registry-tls" not found
Sep 16 02:19:03.752224 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:03.752069    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls podName:7fe7d06d-b82f-407c-9b61-73189c034507 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:11.752063008 +0000 UTC m=+51.182124372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls") pod "dns-default-dcckb" (UID: "7fe7d06d-b82f-407c-9b61-73189c034507") : secret "dns-default-metrics-tls" not found
Sep 16 02:19:04.560127 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:04.560085    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:19:04.560283 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.560243    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 02:19:04.560335 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.560324    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls podName:8fb91d7c-c3a9-40f3-89fa-8aac191a07ef nodeName:}" failed. No retries permitted until 2026-09-16 02:19:12.560307634 +0000 UTC m=+51.990368998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4skl4" (UID: "8fb91d7c-c3a9-40f3-89fa-8aac191a07ef") : secret "cluster-monitoring-operator-tls" not found
Sep 16 02:19:04.660673 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:04.660627    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:04.660883 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:04.660738    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:04.660883 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:04.660767    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:19:04.660883 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.660797    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 02:19:04.661047 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.660889    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:12.660868364 +0000 UTC m=+52.090929728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : secret "router-metrics-certs-default" not found
Sep 16 02:19:04.661047 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.660902    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 02:19:04.661047 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.660905    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:12.660899078 +0000 UTC m=+52.090960442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : configmap references non-existent config key: service-ca.crt
Sep 16 02:19:04.661047 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.661017    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls podName:27b8d016-9177-472c-bb29-5a71c0a14784 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:12.661000874 +0000 UTC m=+52.091062257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-t8w94" (UID: "27b8d016-9177-472c-bb29-5a71c0a14784") : secret "samples-operator-tls" not found
Sep 16 02:19:04.761437 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:04.761380    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:19:04.761859 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.761542    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 02:19:04.761859 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:04.761617    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert podName:512c9eff-f1e7-4507-9d75-fcaf212112e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:12.761599077 +0000 UTC m=+52.191660446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rnf47" (UID: "512c9eff-f1e7-4507-9d75-fcaf212112e4") : secret "networking-console-plugin-cert" not found
Sep 16 02:19:07.443358 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.443317    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x" event={"ID":"4d7b4f33-5eee-47b6-ab81-229546cf3383","Type":"ContainerStarted","Data":"dafca619fbf3f1f6d0de05580d4c268eee77733562264aa02b3ffe7b4eb875ff"}
Sep 16 02:19:07.444993 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.444967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl" event={"ID":"36e5df43-16b6-4495-8396-ac2cc0c0bd0a","Type":"ContainerStarted","Data":"223f2d3d9606d2377daf3c07de551fde44c5269dd7d4ad11324ac1549a09ff4f"}
Sep 16 02:19:07.451985 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.451947    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hs4wt" event={"ID":"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42","Type":"ContainerStarted","Data":"80882e42589962afdee575fa2bdbdd6d3b68f2293fe9ab49203f570856318a03"}
Sep 16 02:19:07.455026 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.455002    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c014add-556b-46aa-a326-69b9bad2ed52" containerID="15a9313f484cd7a7dc167b7ee999d45bddf2046a0589c4b49cd3c5a2568797bc" exitCode=0
Sep 16 02:19:07.455149 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.455084    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerDied","Data":"15a9313f484cd7a7dc167b7ee999d45bddf2046a0589c4b49cd3c5a2568797bc"}
Sep 16 02:19:07.456950 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.456931    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/0.log"
Sep 16 02:19:07.457060 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.456970    2556 generic.go:356] "Generic (PLEG): container finished" podID="97829f87-c6a8-4a47-a358-df8b7eb261dd" containerID="2a3688daf33f37ec2fb4009a8f355ddfa89d231fe3a334e9f7b7d40fffccdb0a" exitCode=255
Sep 16 02:19:07.457186 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.457167    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" event={"ID":"97829f87-c6a8-4a47-a358-df8b7eb261dd","Type":"ContainerDied","Data":"2a3688daf33f37ec2fb4009a8f355ddfa89d231fe3a334e9f7b7d40fffccdb0a"}
Sep 16 02:19:07.457338 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.457261    2556 scope.go:117] "RemoveContainer" containerID="2a3688daf33f37ec2fb4009a8f355ddfa89d231fe3a334e9f7b7d40fffccdb0a"
Sep 16 02:19:07.458619 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.458574    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-4d79x" podStartSLOduration=2.27663272 podStartE2EDuration="11.458558086s" totalImagesPullingTime="9.181925366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:07.456652411 +0000 UTC m=+46.886713799" watchObservedRunningTime="2026-09-16 02:19:07.458558086 +0000 UTC m=+46.888619473"
Sep 16 02:19:07.459891 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.459867    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8" event={"ID":"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2","Type":"ContainerStarted","Data":"dae260cf8a8b454ca207f1569aac64e1493bb22adb86a65f0e2321830b586d7e"}
Sep 16 02:19:07.462068 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.462042    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p" event={"ID":"d29dc62f-9e58-48a5-b87f-074fe8f27b44","Type":"ContainerStarted","Data":"5c7018d30e225ba302debf952a2057c0d6450df4863eb36f7834d4e75d5e6db0"}
Sep 16 02:19:07.463572 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.463554    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gpkxq" event={"ID":"7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6","Type":"ContainerStarted","Data":"bdc2785163affb80919572bd135fc00467d2ff9e21e07316659712f4410e258c"}
Sep 16 02:19:07.470762 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.470702    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-hs4wt" podStartSLOduration=2.376302032 podStartE2EDuration="11.470684377s" totalImagesPullingTime="9.094382345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:07.470420513 +0000 UTC m=+46.900481902" watchObservedRunningTime="2026-09-16 02:19:07.470684377 +0000 UTC m=+46.900745760"
Sep 16 02:19:07.486483 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.486355    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl" podStartSLOduration=2.376675747 podStartE2EDuration="11.486337786s" totalImagesPullingTime="9.109662039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:07.486195641 +0000 UTC m=+46.916257029" watchObservedRunningTime="2026-09-16 02:19:07.486337786 +0000 UTC m=+46.916399173"
Sep 16 02:19:07.535523 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.535466    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gpkxq" podStartSLOduration=2.610493231 podStartE2EDuration="12.53544223s" totalImagesPullingTime="9.924948999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:07.533792623 +0000 UTC m=+46.963854010" watchObservedRunningTime="2026-09-16 02:19:07.53544223 +0000 UTC m=+46.965503608"
Sep 16 02:19:07.559021 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.558966    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8qh6p" podStartSLOduration=2.352713529 podStartE2EDuration="11.558942272s" totalImagesPullingTime="9.206228743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:07.554998474 +0000 UTC m=+46.985059860" watchObservedRunningTime="2026-09-16 02:19:07.558942272 +0000 UTC m=+46.989003659"
Sep 16 02:19:07.568734 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.568702    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"]
Sep 16 02:19:07.580794 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.580752    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8" podStartSLOduration=2.421836512 podStartE2EDuration="11.580738653s" totalImagesPullingTime="9.158902141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:07.580506521 +0000 UTC m=+47.010567910" watchObservedRunningTime="2026-09-16 02:19:07.580738653 +0000 UTC m=+47.010800042"
Sep 16 02:19:07.605954 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.605826    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"]
Sep 16 02:19:07.605954 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.605952    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"
Sep 16 02:19:07.609239 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.608732    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-99tr4\""
Sep 16 02:19:07.609239 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.609049    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 02:19:07.609391 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.609249    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 02:19:07.695619 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.695585    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvvv\" (UniqueName: \"kubernetes.io/projected/52249ffa-3ecf-443a-a076-1f1cdbab07cc-kube-api-access-hjvvv\") pod \"migrator-8b4d697c5-vhp4p\" (UID: \"52249ffa-3ecf-443a-a076-1f1cdbab07cc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"
Sep 16 02:19:07.796257 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.796172    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvvv\" (UniqueName: \"kubernetes.io/projected/52249ffa-3ecf-443a-a076-1f1cdbab07cc-kube-api-access-hjvvv\") pod \"migrator-8b4d697c5-vhp4p\" (UID: \"52249ffa-3ecf-443a-a076-1f1cdbab07cc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"
Sep 16 02:19:07.804802 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.804775    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvvv\" (UniqueName: \"kubernetes.io/projected/52249ffa-3ecf-443a-a076-1f1cdbab07cc-kube-api-access-hjvvv\") pod \"migrator-8b4d697c5-vhp4p\" (UID: \"52249ffa-3ecf-443a-a076-1f1cdbab07cc\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"
Sep 16 02:19:07.930292 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.930255    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"
Sep 16 02:19:07.983777 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:07.983741    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zrb6d"]
Sep 16 02:19:08.021163 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.021130    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zrb6d"]
Sep 16 02:19:08.021309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.021256    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.025054 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.023526    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 02:19:08.025054 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.024031    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74plt\""
Sep 16 02:19:08.025054 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.024233    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 02:19:08.100342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.100262    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.100342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.100314    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjlg\" (UniqueName: \"kubernetes.io/projected/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-api-access-vwjlg\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.100552 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.100462    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-data-volume\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.100552 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.100514    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.100619 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.100551    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-crio-socket\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.201932 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.201896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.202137 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.201948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjlg\" (UniqueName: \"kubernetes.io/projected/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-api-access-vwjlg\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.202137 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.202059    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:08.202137 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.202121    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-data-volume\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.202289 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.202137    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:08.702118827 +0000 UTC m=+48.132180205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:08.202289 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.202178    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.202289 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.202215    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-crio-socket\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.202470 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.202294    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:19:08.202470 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.202344    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:08.702327863 +0000 UTC m=+48.132389228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : configmap "kube-rbac-proxy" not found
Sep 16 02:19:08.202470 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.202424    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-crio-socket\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.202622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.202514    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-data-volume\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.211317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.211292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjlg\" (UniqueName: \"kubernetes.io/projected/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-api-access-vwjlg\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.233264 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.233242    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p"]
Sep 16 02:19:08.236121 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:08.236092    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52249ffa_3ecf_443a_a076_1f1cdbab07cc.slice/crio-f95f7c26622c62f35f866b6196e060dc9a8fa1dbde6bc4e2f404d6cee4c76911 WatchSource:0}: Error finding container f95f7c26622c62f35f866b6196e060dc9a8fa1dbde6bc4e2f404d6cee4c76911: Status 404 returned error can't find the container with id f95f7c26622c62f35f866b6196e060dc9a8fa1dbde6bc4e2f404d6cee4c76911
Sep 16 02:19:08.471513 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.471476    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p" event={"ID":"52249ffa-3ecf-443a-a076-1f1cdbab07cc","Type":"ContainerStarted","Data":"f95f7c26622c62f35f866b6196e060dc9a8fa1dbde6bc4e2f404d6cee4c76911"}
Sep 16 02:19:08.475089 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.475057    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c014add-556b-46aa-a326-69b9bad2ed52" containerID="e5c6d073f20eefdb35fa24f0f4274c354d019608e6779266be9c96eaa1c1c77d" exitCode=0
Sep 16 02:19:08.475336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.475110    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerDied","Data":"e5c6d073f20eefdb35fa24f0f4274c354d019608e6779266be9c96eaa1c1c77d"}
Sep 16 02:19:08.477143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.477124    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:19:08.477631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.477615    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/0.log"
Sep 16 02:19:08.477737 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.477651    2556 generic.go:356] "Generic (PLEG): container finished" podID="97829f87-c6a8-4a47-a358-df8b7eb261dd" containerID="d1c3bfa6bd981388d88674c6f8bd7a1a0c77848ea1eb9b9d098510fd6e09e96b" exitCode=255
Sep 16 02:19:08.477737 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.477706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" event={"ID":"97829f87-c6a8-4a47-a358-df8b7eb261dd","Type":"ContainerDied","Data":"d1c3bfa6bd981388d88674c6f8bd7a1a0c77848ea1eb9b9d098510fd6e09e96b"}
Sep 16 02:19:08.488120 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.488098    2556 scope.go:117] "RemoveContainer" containerID="2a3688daf33f37ec2fb4009a8f355ddfa89d231fe3a334e9f7b7d40fffccdb0a"
Sep 16 02:19:08.488228 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.488206    2556 scope.go:117] "RemoveContainer" containerID="d1c3bfa6bd981388d88674c6f8bd7a1a0c77848ea1eb9b9d098510fd6e09e96b"
Sep 16 02:19:08.488402 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.488384    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mgds9_openshift-console-operator(97829f87-c6a8-4a47-a358-df8b7eb261dd)\"" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" podUID="97829f87-c6a8-4a47-a358-df8b7eb261dd"
Sep 16 02:19:08.707889 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.707848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.708034 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:08.707967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:08.708034 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.708004    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:08.708120 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.708036    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:19:08.708120 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.708091    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:09.708072138 +0000 UTC m=+49.138133520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:08.708120 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:08.708105    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:09.708098711 +0000 UTC m=+49.138160080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : configmap "kube-rbac-proxy" not found
Sep 16 02:19:09.485765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:09.485728    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdff9" event={"ID":"2c014add-556b-46aa-a326-69b9bad2ed52","Type":"ContainerStarted","Data":"d0001ff6b7fcde259f710c54804fa3881648949a680bb92789bcf71919160f54"}
Sep 16 02:19:09.487347 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:09.487323    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:19:09.487629 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:09.487614    2556 scope.go:117] "RemoveContainer" containerID="d1c3bfa6bd981388d88674c6f8bd7a1a0c77848ea1eb9b9d098510fd6e09e96b"
Sep 16 02:19:09.487790 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:09.487773    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mgds9_openshift-console-operator(97829f87-c6a8-4a47-a358-df8b7eb261dd)\"" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" podUID="97829f87-c6a8-4a47-a358-df8b7eb261dd"
Sep 16 02:19:09.509326 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:09.509257    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gdff9" podStartSLOduration=9.552334562 podStartE2EDuration="48.509237161s" totalImagesPullingTime="38.956902599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:09.507002007 +0000 UTC m=+48.937063393" watchObservedRunningTime="2026-09-16 02:19:09.509237161 +0000 UTC m=+48.939298547"
Sep 16 02:19:09.717586 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:09.717551    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:09.717773 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:09.717645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:09.717773 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:09.717722    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:09.717875 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:09.717767    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 02:19:09.717875 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:09.717802    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:11.717780648 +0000 UTC m=+51.147842031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:09.717875 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:09.717824    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:11.717813112 +0000 UTC m=+51.147874476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : configmap "kube-rbac-proxy" not found
Sep 16 02:19:09.779225 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:09.779139    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jn7jj_5b5ea2d3-a6ae-4089-8204-d7422983fe38/dns-node-resolver/0.log"
Sep 16 02:19:10.492316 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.492270    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p" event={"ID":"52249ffa-3ecf-443a-a076-1f1cdbab07cc","Type":"ContainerStarted","Data":"8d2c0a33d2331216769a2da2f9dc81748082f171c473847e3cbe8578b11edfc2"}
Sep 16 02:19:10.492733 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.492372    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p" event={"ID":"52249ffa-3ecf-443a-a076-1f1cdbab07cc","Type":"ContainerStarted","Data":"f39c1d3d7ae0d013e5662a2afd2a4ba1d4b20b43753a1eaa670dc18f64b36cd2"}
Sep 16 02:19:10.507223 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.507168    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vhp4p" podStartSLOduration=2.148720068 podStartE2EDuration="3.507148341s" totalImagesPullingTime="1.358428273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:10.506004108 +0000 UTC m=+49.936065495" watchObservedRunningTime="2026-09-16 02:19:10.507148341 +0000 UTC m=+49.937209736"
Sep 16 02:19:10.845822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.845745    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hdfvz"]
Sep 16 02:19:10.847815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.847797    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:10.851249 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.851224    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pqmm5\""
Sep 16 02:19:10.851372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.851333    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 02:19:10.851585 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.851564    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 02:19:10.852171 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.852153    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 02:19:10.852233 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.852151    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 02:19:10.860963 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.860937    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hdfvz"]
Sep 16 02:19:10.928986 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.928948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sddq\" (UniqueName: \"kubernetes.io/projected/2145c446-e4f5-45b4-8905-1cb6f8569525-kube-api-access-5sddq\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:10.929175 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.929003    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2145c446-e4f5-45b4-8905-1cb6f8569525-signing-key\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:10.929175 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.929100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2145c446-e4f5-45b4-8905-1cb6f8569525-signing-cabundle\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:10.978987 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:10.978959    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9w9ds_6e60ad12-0057-402f-b9fb-56ffa04e13a0/node-ca/0.log"
Sep 16 02:19:11.029530 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.029499    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sddq\" (UniqueName: \"kubernetes.io/projected/2145c446-e4f5-45b4-8905-1cb6f8569525-kube-api-access-5sddq\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:11.029702 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.029555    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2145c446-e4f5-45b4-8905-1cb6f8569525-signing-key\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:11.029702 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.029625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2145c446-e4f5-45b4-8905-1cb6f8569525-signing-cabundle\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:11.030264 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.030241    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2145c446-e4f5-45b4-8905-1cb6f8569525-signing-cabundle\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:11.031907 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.031885    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2145c446-e4f5-45b4-8905-1cb6f8569525-signing-key\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:11.037542 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.037514    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sddq\" (UniqueName: \"kubernetes.io/projected/2145c446-e4f5-45b4-8905-1cb6f8569525-kube-api-access-5sddq\") pod \"service-ca-568869fdc5-hdfvz\" (UID: \"2145c446-e4f5-45b4-8905-1cb6f8569525\") " pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:11.156822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.156740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hdfvz"
Sep 16 02:19:11.425203 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.425166    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hdfvz"]
Sep 16 02:19:11.431638 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:11.431593    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2145c446_e4f5_45b4_8905_1cb6f8569525.slice/crio-1e2d7c709fdd5330c662ba97f35909d456d708936b3c546ca23292658ca7cf80 WatchSource:0}: Error finding container 1e2d7c709fdd5330c662ba97f35909d456d708936b3c546ca23292658ca7cf80: Status 404 returned error can't find the container with id 1e2d7c709fdd5330c662ba97f35909d456d708936b3c546ca23292658ca7cf80
Sep 16 02:19:11.497125 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.497087    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hdfvz" event={"ID":"2145c446-e4f5-45b4-8905-1cb6f8569525","Type":"ContainerStarted","Data":"e02b527d5c5354199be051621d9d4c897c8dd35fd75ccf44a4941ee70d91368a"}
Sep 16 02:19:11.497546 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.497151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hdfvz" event={"ID":"2145c446-e4f5-45b4-8905-1cb6f8569525","Type":"ContainerStarted","Data":"1e2d7c709fdd5330c662ba97f35909d456d708936b3c546ca23292658ca7cf80"}
Sep 16 02:19:11.511466 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.511344    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-hdfvz" podStartSLOduration=1.5113201790000002 podStartE2EDuration="1.511320179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:11.511135973 +0000 UTC m=+50.941197359" watchObservedRunningTime="2026-09-16 02:19:11.511320179 +0000 UTC m=+50.941381566"
Sep 16 02:19:11.580142 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.580108    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gpkxq_7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6/serve-healthcheck-canary/0.log"
Sep 16 02:19:11.735333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.735240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:11.735538 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.735346    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:11.735538 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:11.735462    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:11.735538 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:11.735517    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:15.735503571 +0000 UTC m=+55.165564948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:11.735921 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.735897    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:11.780585 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.780559    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vhp4p_52249ffa-3ecf-443a-a076-1f1cdbab07cc/migrator/0.log"
Sep 16 02:19:11.836039 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.836002    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:19:11.836201 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.836049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:19:11.836201 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:11.836142    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 02:19:11.836275 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:11.836198    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 02:19:11.836275 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:11.836215    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6795b44c76-kkdw4: secret "image-registry-tls" not found
Sep 16 02:19:11.836275 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:11.836202    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls podName:7fe7d06d-b82f-407c-9b61-73189c034507 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:27.836185594 +0000 UTC m=+67.266246979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls") pod "dns-default-dcckb" (UID: "7fe7d06d-b82f-407c-9b61-73189c034507") : secret "dns-default-metrics-tls" not found
Sep 16 02:19:11.836371 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:11.836284    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls podName:c58df75d-dfcd-45eb-8778-e57ae06efc8d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:27.836269704 +0000 UTC m=+67.266331071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls") pod "image-registry-6795b44c76-kkdw4" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d") : secret "image-registry-tls" not found
Sep 16 02:19:11.979642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:11.979613    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vhp4p_52249ffa-3ecf-443a-a076-1f1cdbab07cc/graceful-termination/0.log"
Sep 16 02:19:12.181841 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:12.181809    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5gnj8_dd5bdaad-9ef0-499f-a136-fefde5f6f7d2/kube-storage-version-migrator-operator/0.log"
Sep 16 02:19:12.643843 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:12.643759    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:19:12.644178 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.643929    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 02:19:12.644178 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.644028    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls podName:8fb91d7c-c3a9-40f3-89fa-8aac191a07ef nodeName:}" failed. No retries permitted until 2026-09-16 02:19:28.644006256 +0000 UTC m=+68.074067627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-4skl4" (UID: "8fb91d7c-c3a9-40f3-89fa-8aac191a07ef") : secret "cluster-monitoring-operator-tls" not found
Sep 16 02:19:12.744797 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:12.744745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:12.745008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:12.744875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:12.745008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:12.744908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:19:12.745122 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.745060    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 02:19:12.745178 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.745129    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls podName:27b8d016-9177-472c-bb29-5a71c0a14784 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:28.745107411 +0000 UTC m=+68.175168798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-t8w94" (UID: "27b8d016-9177-472c-bb29-5a71c0a14784") : secret "samples-operator-tls" not found
Sep 16 02:19:12.745483 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.745454    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 02:19:12.745682 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.745463    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:28.745420757 +0000 UTC m=+68.175482135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : configmap references non-existent config key: service-ca.crt
Sep 16 02:19:12.745809 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.745797    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs podName:d501b79f-f7db-41af-93d6-ae81d067cb5d nodeName:}" failed. No retries permitted until 2026-09-16 02:19:28.745779586 +0000 UTC m=+68.175840954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs") pod "router-default-9bc99d698-k2qmx" (UID: "d501b79f-f7db-41af-93d6-ae81d067cb5d") : secret "router-metrics-certs-default" not found
Sep 16 02:19:12.846489 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:12.846444    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:19:12.846666 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.846570    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 02:19:12.846666 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:12.846652    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert podName:512c9eff-f1e7-4507-9d75-fcaf212112e4 nodeName:}" failed. No retries permitted until 2026-09-16 02:19:28.846631931 +0000 UTC m=+68.276693311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-rnf47" (UID: "512c9eff-f1e7-4507-9d75-fcaf212112e4") : secret "networking-console-plugin-cert" not found
Sep 16 02:19:15.774421 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:15.774322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:15.774421 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:15.774416    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:15.774929 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:15.774512    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls podName:2a5cf79b-75fe-4673-8c10-7ef8c1499fbe nodeName:}" failed. No retries permitted until 2026-09-16 02:19:23.774494363 +0000 UTC m=+63.204555727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrb6d" (UID: "2a5cf79b-75fe-4673-8c10-7ef8c1499fbe") : secret "insights-runtime-extractor-tls" not found
Sep 16 02:19:17.238159 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:17.238121    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:19:17.238656 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:17.238186    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:19:17.238656 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:17.238648    2556 scope.go:117] "RemoveContainer" containerID="d1c3bfa6bd981388d88674c6f8bd7a1a0c77848ea1eb9b9d098510fd6e09e96b"
Sep 16 02:19:17.238875 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:19:17.238851    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mgds9_openshift-console-operator(97829f87-c6a8-4a47-a358-df8b7eb261dd)\"" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" podUID="97829f87-c6a8-4a47-a358-df8b7eb261dd"
Sep 16 02:19:20.377035 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:20.377007    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8qvd"
Sep 16 02:19:23.851210 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:23.851164    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:23.853488 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:23.853467    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2a5cf79b-75fe-4673-8c10-7ef8c1499fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrb6d\" (UID: \"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe\") " pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:23.938530 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:23.938499    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74plt\""
Sep 16 02:19:23.947188 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:23.947160    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zrb6d"
Sep 16 02:19:24.230306 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:24.230258    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zrb6d"]
Sep 16 02:19:24.232735 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:24.232704    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5cf79b_75fe_4673_8c10_7ef8c1499fbe.slice/crio-08c9d3f0daa381e49de7eb89d20de0d33df328a0b5f188837304a41a13d73078 WatchSource:0}: Error finding container 08c9d3f0daa381e49de7eb89d20de0d33df328a0b5f188837304a41a13d73078: Status 404 returned error can't find the container with id 08c9d3f0daa381e49de7eb89d20de0d33df328a0b5f188837304a41a13d73078
Sep 16 02:19:24.545837 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:24.545753    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrb6d" event={"ID":"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe","Type":"ContainerStarted","Data":"c623cdf5c6f2224b0a2ce22754237838dbc0a8023b2bd3ab341c276840c9dd48"}
Sep 16 02:19:24.545837 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:24.545836    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrb6d" event={"ID":"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe","Type":"ContainerStarted","Data":"08c9d3f0daa381e49de7eb89d20de0d33df328a0b5f188837304a41a13d73078"}
Sep 16 02:19:25.551760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:25.551727    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrb6d" event={"ID":"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe","Type":"ContainerStarted","Data":"343995b6c4f9438935d087a618719e0d1f6aed40fa1be3fd13c5740f8cb270bc"}
Sep 16 02:19:26.678302 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.678267    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:19:26.680971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.680669    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 02:19:26.690791 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.690763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103d25bf-1d27-4cb9-be5c-cf6a8f8e691b-metrics-certs\") pod \"network-metrics-daemon-t4vxs\" (UID: \"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b\") " pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:19:26.778724 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.778686    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:19:26.781336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.781316    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 02:19:26.791397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.791376    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b83e697-837d-4c0e-b454-767d6743f7ab-original-pull-secret\") pod \"global-pull-secret-syncer-plfnq\" (UID: \"9b83e697-837d-4c0e-b454-767d6743f7ab\") " pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:19:26.848223 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.848191    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ss74q\""
Sep 16 02:19:26.855286 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.855255    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfnq"
Sep 16 02:19:26.856322 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.856304    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4vxs"
Sep 16 02:19:26.879506 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.879478    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:19:26.881922 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:26.881896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62w8z\" (UniqueName: \"kubernetes.io/projected/3b94c46a-de62-4b93-ba0c-66059c365abb-kube-api-access-62w8z\") pod \"network-check-target-4sfbv\" (UID: \"3b94c46a-de62-4b93-ba0c-66059c365abb\") " pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:19:27.129385 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.129353    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4vxs"]
Sep 16 02:19:27.131765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.131738    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plfnq"]
Sep 16 02:19:27.132712 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:27.132579    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103d25bf_1d27_4cb9_be5c_cf6a8f8e691b.slice/crio-44ba3f265177a8a0f449c912fca3f711c0aa366f5cefa2fd1c3fb2d22671ec46 WatchSource:0}: Error finding container 44ba3f265177a8a0f449c912fca3f711c0aa366f5cefa2fd1c3fb2d22671ec46: Status 404 returned error can't find the container with id 44ba3f265177a8a0f449c912fca3f711c0aa366f5cefa2fd1c3fb2d22671ec46
Sep 16 02:19:27.134556 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:27.134409    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b83e697_837d_4c0e_b454_767d6743f7ab.slice/crio-1b8ea1c8f46f6934e0c094922b0fd244ca49883395648a64aa7c1a98fe122d33 WatchSource:0}: Error finding container 1b8ea1c8f46f6934e0c094922b0fd244ca49883395648a64aa7c1a98fe122d33: Status 404 returned error can't find the container with id 1b8ea1c8f46f6934e0c094922b0fd244ca49883395648a64aa7c1a98fe122d33
Sep 16 02:19:27.135758 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.135739    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7ktl5\""
Sep 16 02:19:27.143816 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.143766    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:19:27.412347 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.412314    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4sfbv"]
Sep 16 02:19:27.414998 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:27.414969    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b94c46a_de62_4b93_ba0c_66059c365abb.slice/crio-21ac343ad89f3742a4ae45834c27c0ea297fdd1052dff9aa933eb31edeffb310 WatchSource:0}: Error finding container 21ac343ad89f3742a4ae45834c27c0ea297fdd1052dff9aa933eb31edeffb310: Status 404 returned error can't find the container with id 21ac343ad89f3742a4ae45834c27c0ea297fdd1052dff9aa933eb31edeffb310
Sep 16 02:19:27.560498 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.560454    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4vxs" event={"ID":"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b","Type":"ContainerStarted","Data":"44ba3f265177a8a0f449c912fca3f711c0aa366f5cefa2fd1c3fb2d22671ec46"}
Sep 16 02:19:27.562845 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.562809    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrb6d" event={"ID":"2a5cf79b-75fe-4673-8c10-7ef8c1499fbe","Type":"ContainerStarted","Data":"551c106f8d2d0422cc0a4da0e43dc46bbd7563b5aa4c5174bf88ab134a41cd18"}
Sep 16 02:19:27.564371 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.564344    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4sfbv" event={"ID":"3b94c46a-de62-4b93-ba0c-66059c365abb","Type":"ContainerStarted","Data":"82e20659cabf704c29526f127442bc411051e1b39836f272d2b71d2e2297b85b"}
Sep 16 02:19:27.564514 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.564389    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:19:27.564514 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.564403    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4sfbv" event={"ID":"3b94c46a-de62-4b93-ba0c-66059c365abb","Type":"ContainerStarted","Data":"21ac343ad89f3742a4ae45834c27c0ea297fdd1052dff9aa933eb31edeffb310"}
Sep 16 02:19:27.565520 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.565486    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plfnq" event={"ID":"9b83e697-837d-4c0e-b454-767d6743f7ab","Type":"ContainerStarted","Data":"1b8ea1c8f46f6934e0c094922b0fd244ca49883395648a64aa7c1a98fe122d33"}
Sep 16 02:19:27.579984 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.579927    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zrb6d" podStartSLOduration=18.270022867 podStartE2EDuration="20.579905646s" totalImagesPullingTime="2.309882779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:19:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:27.578330001 +0000 UTC m=+67.008391385" watchObservedRunningTime="2026-09-16 02:19:27.579905646 +0000 UTC m=+67.009967034"
Sep 16 02:19:27.591193 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.591137    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4sfbv" podStartSLOduration=66.591114682 podStartE2EDuration="1m6.591114682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:27.590478284 +0000 UTC m=+67.020539672" watchObservedRunningTime="2026-09-16 02:19:27.591114682 +0000 UTC m=+67.021176069"
Sep 16 02:19:27.888184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.888092    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:19:27.888184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.888149    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:19:27.891108 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.891060    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"image-registry-6795b44c76-kkdw4\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") " pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:19:27.891322 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:27.891298    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7fe7d06d-b82f-407c-9b61-73189c034507-metrics-tls\") pod \"dns-default-dcckb\" (UID: \"7fe7d06d-b82f-407c-9b61-73189c034507\") " pod="openshift-dns/dns-default-dcckb"
Sep 16 02:19:28.054206 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.054174    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4mjd\""
Sep 16 02:19:28.062939 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.062905    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:19:28.068419 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.068401    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w4x8g\""
Sep 16 02:19:28.077308 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.077281    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dcckb"
Sep 16 02:19:28.338822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.338798    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6795b44c76-kkdw4"]
Sep 16 02:19:28.342600 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:28.342443    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58df75d_dfcd_45eb_8778_e57ae06efc8d.slice/crio-af196abc082726f5dc4e9768a36206d456c6fa22f9d645314d2e8574123f07c1 WatchSource:0}: Error finding container af196abc082726f5dc4e9768a36206d456c6fa22f9d645314d2e8574123f07c1: Status 404 returned error can't find the container with id af196abc082726f5dc4e9768a36206d456c6fa22f9d645314d2e8574123f07c1
Sep 16 02:19:28.391909 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.391818    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dcckb"]
Sep 16 02:19:28.393904 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:28.393882    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe7d06d_b82f_407c_9b61_73189c034507.slice/crio-0b692bcc7b098dc250501e26ddecdf1c970dbb76722964bd6ba80c994d580cb7 WatchSource:0}: Error finding container 0b692bcc7b098dc250501e26ddecdf1c970dbb76722964bd6ba80c994d580cb7: Status 404 returned error can't find the container with id 0b692bcc7b098dc250501e26ddecdf1c970dbb76722964bd6ba80c994d580cb7
Sep 16 02:19:28.577563 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.577453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcckb" event={"ID":"7fe7d06d-b82f-407c-9b61-73189c034507","Type":"ContainerStarted","Data":"0b692bcc7b098dc250501e26ddecdf1c970dbb76722964bd6ba80c994d580cb7"}
Sep 16 02:19:28.579321 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.579292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4" event={"ID":"c58df75d-dfcd-45eb-8778-e57ae06efc8d","Type":"ContainerStarted","Data":"f4f28cba1c394cd140ca90ebfef00b686a3b64a2f1bccc82684d0e389adbb4f9"}
Sep 16 02:19:28.579477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.579359    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4" event={"ID":"c58df75d-dfcd-45eb-8778-e57ae06efc8d","Type":"ContainerStarted","Data":"af196abc082726f5dc4e9768a36206d456c6fa22f9d645314d2e8574123f07c1"}
Sep 16 02:19:28.579652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.579629    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:19:28.581551 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.581522    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4vxs" event={"ID":"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b","Type":"ContainerStarted","Data":"9fc7b516012277704bf40dc2760ac06af45088d9e1772af743e720502cabbf9c"}
Sep 16 02:19:28.581661 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.581579    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4vxs" event={"ID":"103d25bf-1d27-4cb9-be5c-cf6a8f8e691b","Type":"ContainerStarted","Data":"ebafcd996bd9782f18a8654f6d9f7aef07616c812c4e0be2b015b321f44412f9"}
Sep 16 02:19:28.598440 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.598382    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4" podStartSLOduration=67.59836351 podStartE2EDuration="1m7.59836351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:28.596671063 +0000 UTC m=+68.026732454" watchObservedRunningTime="2026-09-16 02:19:28.59836351 +0000 UTC m=+68.028424956"
Sep 16 02:19:28.611511 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.611454    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t4vxs" podStartSLOduration=66.628030334 podStartE2EDuration="1m7.611418586s" totalImagesPullingTime="983.388252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:28.609149837 +0000 UTC m=+68.039211224" watchObservedRunningTime="2026-09-16 02:19:28.611418586 +0000 UTC m=+68.041479981"
Sep 16 02:19:28.697116 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.696574    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:19:28.699706 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.699679    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fb91d7c-c3a9-40f3-89fa-8aac191a07ef-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-4skl4\" (UID: \"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:19:28.798125 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.798081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:28.798358 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.798194    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:28.798358 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.798226    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:19:28.799008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.798938    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d501b79f-f7db-41af-93d6-ae81d067cb5d-service-ca-bundle\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:28.801012 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.800982    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d501b79f-f7db-41af-93d6-ae81d067cb5d-metrics-certs\") pod \"router-default-9bc99d698-k2qmx\" (UID: \"d501b79f-f7db-41af-93d6-ae81d067cb5d\") " pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:28.801127 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.801072    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/27b8d016-9177-472c-bb29-5a71c0a14784-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-t8w94\" (UID: \"27b8d016-9177-472c-bb29-5a71c0a14784\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:19:28.899044 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.898958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:19:28.901848 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.901822    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/512c9eff-f1e7-4507-9d75-fcaf212112e4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rnf47\" (UID: \"512c9eff-f1e7-4507-9d75-fcaf212112e4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:19:28.923708 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.923680    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dpl6r\""
Sep 16 02:19:28.931933 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.931907    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"
Sep 16 02:19:28.976224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.976135    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tnzdb\""
Sep 16 02:19:28.984171 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.984135    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"
Sep 16 02:19:28.999934 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:28.999832    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8rl2z\""
Sep 16 02:19:29.008002 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.007973    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:29.028259 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.028045    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8kmhr\""
Sep 16 02:19:29.037193 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.037014    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"
Sep 16 02:19:29.219401 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.219149    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4"]
Sep 16 02:19:29.222494 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:29.222461    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb91d7c_c3a9_40f3_89fa_8aac191a07ef.slice/crio-dd4eee3a2717f8d19375246d667d949a21a95e92d99fc2004a0d93bb264da0b6 WatchSource:0}: Error finding container dd4eee3a2717f8d19375246d667d949a21a95e92d99fc2004a0d93bb264da0b6: Status 404 returned error can't find the container with id dd4eee3a2717f8d19375246d667d949a21a95e92d99fc2004a0d93bb264da0b6
Sep 16 02:19:29.276604 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.276522    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94"]
Sep 16 02:19:29.304538 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.304512    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9bc99d698-k2qmx"]
Sep 16 02:19:29.312849 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:29.312811    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd501b79f_f7db_41af_93d6_ae81d067cb5d.slice/crio-ad232da788c6fa03c54cd2c68bffe06dad7c3665edc4a24fbfbcb4317d234771 WatchSource:0}: Error finding container ad232da788c6fa03c54cd2c68bffe06dad7c3665edc4a24fbfbcb4317d234771: Status 404 returned error can't find the container with id ad232da788c6fa03c54cd2c68bffe06dad7c3665edc4a24fbfbcb4317d234771
Sep 16 02:19:29.336593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.336344    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47"]
Sep 16 02:19:29.341306 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:29.341273    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512c9eff_f1e7_4507_9d75_fcaf212112e4.slice/crio-70e624d77af351716b411c14be2bec7acdb96efc548ff965b427ae578fec1d2e WatchSource:0}: Error finding container 70e624d77af351716b411c14be2bec7acdb96efc548ff965b427ae578fec1d2e: Status 404 returned error can't find the container with id 70e624d77af351716b411c14be2bec7acdb96efc548ff965b427ae578fec1d2e
Sep 16 02:19:29.587417 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.587325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47" event={"ID":"512c9eff-f1e7-4507-9d75-fcaf212112e4","Type":"ContainerStarted","Data":"70e624d77af351716b411c14be2bec7acdb96efc548ff965b427ae578fec1d2e"}
Sep 16 02:19:29.588780 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.588745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4" event={"ID":"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef","Type":"ContainerStarted","Data":"dd4eee3a2717f8d19375246d667d949a21a95e92d99fc2004a0d93bb264da0b6"}
Sep 16 02:19:29.590483 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.590455    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9bc99d698-k2qmx" event={"ID":"d501b79f-f7db-41af-93d6-ae81d067cb5d","Type":"ContainerStarted","Data":"c5a298cbce6c180a14b5073af6cd78cc4ea30736fd248ef35b98b992c8662647"}
Sep 16 02:19:29.590593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.590504    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9bc99d698-k2qmx" event={"ID":"d501b79f-f7db-41af-93d6-ae81d067cb5d","Type":"ContainerStarted","Data":"ad232da788c6fa03c54cd2c68bffe06dad7c3665edc4a24fbfbcb4317d234771"}
Sep 16 02:19:29.592020 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.591989    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94" event={"ID":"27b8d016-9177-472c-bb29-5a71c0a14784","Type":"ContainerStarted","Data":"c88c02716ce717f7eacbdb23e1d53aa7368109b57f4e401f32650311deacd2a9"}
Sep 16 02:19:29.607967 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:29.607910    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-9bc99d698-k2qmx" podStartSLOduration=33.607888528 podStartE2EDuration="33.607888528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:29.606709949 +0000 UTC m=+69.036771345" watchObservedRunningTime="2026-09-16 02:19:29.607888528 +0000 UTC m=+69.037949914"
Sep 16 02:19:30.008459 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:30.008412    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:30.011629 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:30.011400    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:30.596230 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:30.596165    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:30.597731 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:30.597700    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-9bc99d698-k2qmx"
Sep 16 02:19:32.108265 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:32.108229    2556 scope.go:117] "RemoveContainer" containerID="d1c3bfa6bd981388d88674c6f8bd7a1a0c77848ea1eb9b9d098510fd6e09e96b"
Sep 16 02:19:33.045169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.045116    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"]
Sep 16 02:19:33.068708 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.068139    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"]
Sep 16 02:19:33.069719 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.069635    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.073748 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.073710    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 02:19:33.073894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.073819    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 02:19:33.073986 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.073967    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 02:19:33.074598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.074575    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 02:19:33.081256 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.081086    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"]
Sep 16 02:19:33.081256 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.081212    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.084261 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.084082    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 02:19:33.084261 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.084140    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5lpml\""
Sep 16 02:19:33.095707 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.095633    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"]
Sep 16 02:19:33.095707 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.095662    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"]
Sep 16 02:19:33.095707 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.095693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"]
Sep 16 02:19:33.095929 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.095798    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.098597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.098573    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 02:19:33.098769 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.098745    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 02:19:33.098879 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.098749    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 02:19:33.099234 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.099216    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 02:19:33.237816 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.237778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5gvf\" (UniqueName: \"kubernetes.io/projected/fc1c1003-e5cb-4958-ae97-8984cd818a42-kube-api-access-b5gvf\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.237823    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-hub\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.237876    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.237966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76af2359-7728-43a2-a9d0-94f24055bd2f-tmp\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.238024    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c19f192-a93f-46ef-8134-b69ce1903600-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66d8969bd6-dtprj\" (UID: \"9c19f192-a93f-46ef-8134-b69ce1903600\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.238054    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwq6\" (UniqueName: \"kubernetes.io/projected/76af2359-7728-43a2-a9d0-94f24055bd2f-kube-api-access-kwwq6\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.238072    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-ca\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.238087    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.238139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/76af2359-7728-43a2-a9d0-94f24055bd2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.238182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.238173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c1003-e5cb-4958-ae97-8984cd818a42-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.238516 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.238195    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssl2x\" (UniqueName: \"kubernetes.io/projected/9c19f192-a93f-46ef-8134-b69ce1903600-kube-api-access-ssl2x\") pod \"managed-serviceaccount-addon-agent-66d8969bd6-dtprj\" (UID: \"9c19f192-a93f-46ef-8134-b69ce1903600\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.339223 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76af2359-7728-43a2-a9d0-94f24055bd2f-tmp\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.339398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c19f192-a93f-46ef-8134-b69ce1903600-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66d8969bd6-dtprj\" (UID: \"9c19f192-a93f-46ef-8134-b69ce1903600\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.339398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339258    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwq6\" (UniqueName: \"kubernetes.io/projected/76af2359-7728-43a2-a9d0-94f24055bd2f-kube-api-access-kwwq6\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.339398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-ca\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.339398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339303    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.339398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/76af2359-7728-43a2-a9d0-94f24055bd2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.339398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339363    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c1003-e5cb-4958-ae97-8984cd818a42-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.339398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssl2x\" (UniqueName: \"kubernetes.io/projected/9c19f192-a93f-46ef-8134-b69ce1903600-kube-api-access-ssl2x\") pod \"managed-serviceaccount-addon-agent-66d8969bd6-dtprj\" (UID: \"9c19f192-a93f-46ef-8134-b69ce1903600\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.339764 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5gvf\" (UniqueName: \"kubernetes.io/projected/fc1c1003-e5cb-4958-ae97-8984cd818a42-kube-api-access-b5gvf\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.339764 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339488    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-hub\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.339764 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.339764 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.339620    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76af2359-7728-43a2-a9d0-94f24055bd2f-tmp\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.340317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.340281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c1003-e5cb-4958-ae97-8984cd818a42-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.342419 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.342384    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-ca\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.342550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.342495    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.342550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.342517    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-hub\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.342631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.342585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/76af2359-7728-43a2-a9d0-94f24055bd2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.343016 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.342987    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c19f192-a93f-46ef-8134-b69ce1903600-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66d8969bd6-dtprj\" (UID: \"9c19f192-a93f-46ef-8134-b69ce1903600\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.343093 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.343030    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fc1c1003-e5cb-4958-ae97-8984cd818a42-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.350584 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.350559    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5gvf\" (UniqueName: \"kubernetes.io/projected/fc1c1003-e5cb-4958-ae97-8984cd818a42-kube-api-access-b5gvf\") pod \"cluster-proxy-proxy-agent-7c69999f95-gpjsq\" (UID: \"fc1c1003-e5cb-4958-ae97-8984cd818a42\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:33.350915 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.350855    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwq6\" (UniqueName: \"kubernetes.io/projected/76af2359-7728-43a2-a9d0-94f24055bd2f-kube-api-access-kwwq6\") pod \"klusterlet-addon-workmgr-7c695c86f5-ht8mf\" (UID: \"76af2359-7728-43a2-a9d0-94f24055bd2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.351312 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.351295    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssl2x\" (UniqueName: \"kubernetes.io/projected/9c19f192-a93f-46ef-8134-b69ce1903600-kube-api-access-ssl2x\") pod \"managed-serviceaccount-addon-agent-66d8969bd6-dtprj\" (UID: \"9c19f192-a93f-46ef-8134-b69ce1903600\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.384532 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.384508    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:33.403562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.403542    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"
Sep 16 02:19:33.414142 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:33.414036    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"
Sep 16 02:19:34.228061 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.228023    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"]
Sep 16 02:19:34.232821 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.232761    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq"]
Sep 16 02:19:34.244650 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.244068    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj"]
Sep 16 02:19:34.278600 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:34.278562    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76af2359_7728_43a2_a9d0_94f24055bd2f.slice/crio-9fa75d0201870a7fbed371e7067cc9f7a4a0e67095526f20f00efa37ba720507 WatchSource:0}: Error finding container 9fa75d0201870a7fbed371e7067cc9f7a4a0e67095526f20f00efa37ba720507: Status 404 returned error can't find the container with id 9fa75d0201870a7fbed371e7067cc9f7a4a0e67095526f20f00efa37ba720507
Sep 16 02:19:34.279605 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:34.279175    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc1c1003_e5cb_4958_ae97_8984cd818a42.slice/crio-267f57357408cac1a0a323be972423bf746ccaaf5b6726f36005e6ead470ed77 WatchSource:0}: Error finding container 267f57357408cac1a0a323be972423bf746ccaaf5b6726f36005e6ead470ed77: Status 404 returned error can't find the container with id 267f57357408cac1a0a323be972423bf746ccaaf5b6726f36005e6ead470ed77
Sep 16 02:19:34.614105 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.613605    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47" event={"ID":"512c9eff-f1e7-4507-9d75-fcaf212112e4","Type":"ContainerStarted","Data":"2ee4a21b29bb4e0ba46077c8ea7c33894f32f2d39b430be441f81bfaa25f37d7"}
Sep 16 02:19:34.615447 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.615395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4" event={"ID":"8fb91d7c-c3a9-40f3-89fa-8aac191a07ef","Type":"ContainerStarted","Data":"7dca8a4279cb3e44cd53fada3eb2ddd86065f279c2e29291bb5b81d953bee230"}
Sep 16 02:19:34.617306 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.617285    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq" event={"ID":"fc1c1003-e5cb-4958-ae97-8984cd818a42","Type":"ContainerStarted","Data":"267f57357408cac1a0a323be972423bf746ccaaf5b6726f36005e6ead470ed77"}
Sep 16 02:19:34.619384 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.619367    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:19:34.619490 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.619443    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" event={"ID":"97829f87-c6a8-4a47-a358-df8b7eb261dd","Type":"ContainerStarted","Data":"780bb267827dd380a5e2beed6bacacf7a6193d9c4cb7fab236314f635d94b962"}
Sep 16 02:19:34.619892 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.619721    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:19:34.621405 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.621382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plfnq" event={"ID":"9b83e697-837d-4c0e-b454-767d6743f7ab","Type":"ContainerStarted","Data":"70073ce2ab2f615eae16c04b348052e516af6155d1adba8ee046c2e1952ab2d8"}
Sep 16 02:19:34.623355 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.623328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94" event={"ID":"27b8d016-9177-472c-bb29-5a71c0a14784","Type":"ContainerStarted","Data":"b16ead0cc1be9f1e87fe985738055b9bf869932f2c1b9bdbc5621299c9639695"}
Sep 16 02:19:34.623454 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.623380    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94" event={"ID":"27b8d016-9177-472c-bb29-5a71c0a14784","Type":"ContainerStarted","Data":"e8c50d37c56eb2d326f9ba547808bf65c40c2f5eb104fab16fd65b06971c2dbc"}
Sep 16 02:19:34.624714 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.624665    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf" event={"ID":"76af2359-7728-43a2-a9d0-94f24055bd2f","Type":"ContainerStarted","Data":"9fa75d0201870a7fbed371e7067cc9f7a4a0e67095526f20f00efa37ba720507"}
Sep 16 02:19:34.626774 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.626754    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcckb" event={"ID":"7fe7d06d-b82f-407c-9b61-73189c034507","Type":"ContainerStarted","Data":"26b8dabc015b999d08141114179456fe2c37b6d4b5b7624b530b91292eb20bbf"}
Sep 16 02:19:34.626929 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.626914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcckb" event={"ID":"7fe7d06d-b82f-407c-9b61-73189c034507","Type":"ContainerStarted","Data":"eabb136ee7edfb9ca0bf699c3ae5e89051dd8323733b5a9099e8d6c369889cf3"}
Sep 16 02:19:34.627036 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.627024    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dcckb"
Sep 16 02:19:34.628617 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.627709    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rnf47" podStartSLOduration=33.671291209 podStartE2EDuration="38.627668483s" totalImagesPullingTime="4.956377274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:34.627462798 +0000 UTC m=+74.057524186" watchObservedRunningTime="2026-09-16 02:19:34.627668483 +0000 UTC m=+74.057729872"
Sep 16 02:19:34.628617 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.628291    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj" event={"ID":"9c19f192-a93f-46ef-8134-b69ce1903600","Type":"ContainerStarted","Data":"931e9bc8e40035b63cfc0c8072c087e8760be16624087738ce44cbc6a8c1a275"}
Sep 16 02:19:34.693470 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.693386    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-t8w94" podStartSLOduration=34.08662766 podStartE2EDuration="38.693365401s" totalImagesPullingTime="4.606737741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:34.692552707 +0000 UTC m=+74.122614091" watchObservedRunningTime="2026-09-16 02:19:34.693365401 +0000 UTC m=+74.123426789"
Sep 16 02:19:34.694575 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.694534    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-plfnq" podStartSLOduration=66.890082274 podStartE2EDuration="1m13.694517079s" totalImagesPullingTime="6.804434805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:34.643387392 +0000 UTC m=+74.073448776" watchObservedRunningTime="2026-09-16 02:19:34.694517079 +0000 UTC m=+74.124578466"
Sep 16 02:19:34.714832 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.714779    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-4skl4" podStartSLOduration=33.998736145 podStartE2EDuration="38.714762232s" totalImagesPullingTime="4.716026087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:34.714005132 +0000 UTC m=+74.144066520" watchObservedRunningTime="2026-09-16 02:19:34.714762232 +0000 UTC m=+74.144823675"
Sep 16 02:19:34.729883 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.729779    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dcckb" podStartSLOduration=34.180942229 podStartE2EDuration="39.729763008s" totalImagesPullingTime="5.548820779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:34.728609121 +0000 UTC m=+74.158670532" watchObservedRunningTime="2026-09-16 02:19:34.729763008 +0000 UTC m=+74.159824436"
Sep 16 02:19:34.749016 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.748971    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-mgds9" podStartSLOduration=29.729838621 podStartE2EDuration="38.748956893s" totalImagesPullingTime="9.019118272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:34.747865831 +0000 UTC m=+74.177927218" watchObservedRunningTime="2026-09-16 02:19:34.748956893 +0000 UTC m=+74.179018278"
Sep 16 02:19:34.974915 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:34.974884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-mgds9"
Sep 16 02:19:42.669411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.669370    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf" event={"ID":"76af2359-7728-43a2-a9d0-94f24055bd2f","Type":"ContainerStarted","Data":"29aed75eac4462b3b4a0bba0b14906d3e5f4adc0f4b3663106b77a9bd099d61d"}
Sep 16 02:19:42.669889 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.669541    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:42.671024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.670978    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj" event={"ID":"9c19f192-a93f-46ef-8134-b69ce1903600","Type":"ContainerStarted","Data":"dd5b7ef486863511c0356f3e0414b61d6347dce3afc097c74e89dccda28c736a"}
Sep 16 02:19:42.671569 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.671545    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf"
Sep 16 02:19:42.673231 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.673201    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq" event={"ID":"fc1c1003-e5cb-4958-ae97-8984cd818a42","Type":"ContainerStarted","Data":"d0ca11ffc09db9ce1764c77c7f3d24b4f76819f930fcd6798ef5ad0394d3b65e"}
Sep 16 02:19:42.673231 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.673230    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq" event={"ID":"fc1c1003-e5cb-4958-ae97-8984cd818a42","Type":"ContainerStarted","Data":"8aeeee59c33172c7440f24c189786f1fa03eee1af6d97284a4eef59efd8966a6"}
Sep 16 02:19:42.673349 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.673242    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq" event={"ID":"fc1c1003-e5cb-4958-ae97-8984cd818a42","Type":"ContainerStarted","Data":"ce4354c4fa3caced71e18182a55609429b3b4ad80f1e9a82f508a1820c9454be"}
Sep 16 02:19:42.684825 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.684778    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c695c86f5-ht8mf" podStartSLOduration=2.085847972 podStartE2EDuration="9.684764523s" totalImagesPullingTime="7.598916551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:42.684395031 +0000 UTC m=+82.114456418" watchObservedRunningTime="2026-09-16 02:19:42.684764523 +0000 UTC m=+82.114825911"
Sep 16 02:19:42.711395 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.711345    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66d8969bd6-dtprj" podStartSLOduration=2.102850822 podStartE2EDuration="9.711331888s" totalImagesPullingTime="7.608481066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:42.710580011 +0000 UTC m=+82.140641408" watchObservedRunningTime="2026-09-16 02:19:42.711331888 +0000 UTC m=+82.141393291"
Sep 16 02:19:42.728464 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:42.728407    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c69999f95-gpjsq" podStartSLOduration=2.12315511 podStartE2EDuration="9.728397647s" totalImagesPullingTime="7.605242537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:42.727800305 +0000 UTC m=+82.157861692" watchObservedRunningTime="2026-09-16 02:19:42.728397647 +0000 UTC m=+82.158459059"
Sep 16 02:19:44.003024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.002985    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t5q6s"]
Sep 16 02:19:44.006183 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.006160    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.008720 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.008690    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 02:19:44.008838 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.008716    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fkjtp\""
Sep 16 02:19:44.008998 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.008983    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 02:19:44.009104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.009030    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 02:19:44.009928 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.009909    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 02:19:44.129207 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-sys\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129352 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129212    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129352 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129273    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-tls\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129352 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129326    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-wtmp\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129483 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129355    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef70daa3-7579-406f-ab26-ed24cb298c95-metrics-client-ca\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129483 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129418    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-textfile\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129483 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129470    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72trm\" (UniqueName: \"kubernetes.io/projected/ef70daa3-7579-406f-ab26-ed24cb298c95-kube-api-access-72trm\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129570 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129503    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.129570 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.129539    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-root\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.229981 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.229927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-textfile\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.229993    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72trm\" (UniqueName: \"kubernetes.io/projected/ef70daa3-7579-406f-ab26-ed24cb298c95-kube-api-access-72trm\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230022    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230056    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-root\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230104    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-sys\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-tls\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-wtmp\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-sys\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef70daa3-7579-406f-ab26-ed24cb298c95-metrics-client-ca\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-root\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-textfile\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230462    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-wtmp\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230875 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef70daa3-7579-406f-ab26-ed24cb298c95-metrics-client-ca\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.230972 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.230875    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.232551 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.232529    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.232889 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.232868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef70daa3-7579-406f-ab26-ed24cb298c95-node-exporter-tls\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.237603 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.237580    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72trm\" (UniqueName: \"kubernetes.io/projected/ef70daa3-7579-406f-ab26-ed24cb298c95-kube-api-access-72trm\") pod \"node-exporter-t5q6s\" (UID: \"ef70daa3-7579-406f-ab26-ed24cb298c95\") " pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.316398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.316305    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5q6s"
Sep 16 02:19:44.329708 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:19:44.329670    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef70daa3_7579_406f_ab26_ed24cb298c95.slice/crio-3de8580a8af6757922f8edfde1e2ac38e88ddbdd937ff2f5dce22fce816eb0f5 WatchSource:0}: Error finding container 3de8580a8af6757922f8edfde1e2ac38e88ddbdd937ff2f5dce22fce816eb0f5: Status 404 returned error can't find the container with id 3de8580a8af6757922f8edfde1e2ac38e88ddbdd937ff2f5dce22fce816eb0f5
Sep 16 02:19:44.649695 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.649614    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dcckb"
Sep 16 02:19:44.683560 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:44.683524    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5q6s" event={"ID":"ef70daa3-7579-406f-ab26-ed24cb298c95","Type":"ContainerStarted","Data":"3de8580a8af6757922f8edfde1e2ac38e88ddbdd937ff2f5dce22fce816eb0f5"}
Sep 16 02:19:45.689915 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:45.689879    2556 generic.go:356] "Generic (PLEG): container finished" podID="ef70daa3-7579-406f-ab26-ed24cb298c95" containerID="42d096e040545440bbc51e52cb2ba2a5e47e37743a3058aaa38b609d95d35e00" exitCode=0
Sep 16 02:19:45.690359 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:45.689974    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5q6s" event={"ID":"ef70daa3-7579-406f-ab26-ed24cb298c95","Type":"ContainerDied","Data":"42d096e040545440bbc51e52cb2ba2a5e47e37743a3058aaa38b609d95d35e00"}
Sep 16 02:19:46.696956 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:46.696923    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5q6s" event={"ID":"ef70daa3-7579-406f-ab26-ed24cb298c95","Type":"ContainerStarted","Data":"ea6193d6218b230510db0de1d911c6354512cbaf0a603b2f02e412e1c4ed586e"}
Sep 16 02:19:46.697327 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:46.696975    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5q6s" event={"ID":"ef70daa3-7579-406f-ab26-ed24cb298c95","Type":"ContainerStarted","Data":"c69c5aebefe23fdb8f48521c3959eebaee885a258c69eda76354cd73d35200d0"}
Sep 16 02:19:48.067161 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:48.067125    2556 patch_prober.go:28] interesting pod/image-registry-6795b44c76-kkdw4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 02:19:48.067562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:48.067188    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4" podUID="c58df75d-dfcd-45eb-8778-e57ae06efc8d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 02:19:49.596270 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:49.596241    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:19:49.614787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:49.614743    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t5q6s" podStartSLOduration=5.621684294 podStartE2EDuration="6.614727612s" totalImagesPullingTime="928.774422ms" totalInitContainerRuntime="64.268896ms" isStatefulPod=true podCreationTimestamp="2026-09-16 02:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:19:46.72079965 +0000 UTC m=+86.150861037" watchObservedRunningTime="2026-09-16 02:19:49.614727612 +0000 UTC m=+89.044788997"
Sep 16 02:19:55.674267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:55.674235    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6795b44c76-kkdw4"]
Sep 16 02:19:58.584692 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:19:58.584656    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4sfbv"
Sep 16 02:20:20.693656 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:20.693598    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4" podUID="c58df75d-dfcd-45eb-8778-e57ae06efc8d" containerName="registry" containerID="cri-o://f4f28cba1c394cd140ca90ebfef00b686a3b64a2f1bccc82684d0e389adbb4f9" gracePeriod=30
Sep 16 02:20:20.836182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:20.836139    2556 generic.go:356] "Generic (PLEG): container finished" podID="c58df75d-dfcd-45eb-8778-e57ae06efc8d" containerID="f4f28cba1c394cd140ca90ebfef00b686a3b64a2f1bccc82684d0e389adbb4f9" exitCode=0
Sep 16 02:20:20.836360 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:20.836250    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4" event={"ID":"c58df75d-dfcd-45eb-8778-e57ae06efc8d","Type":"ContainerDied","Data":"f4f28cba1c394cd140ca90ebfef00b686a3b64a2f1bccc82684d0e389adbb4f9"}
Sep 16 02:20:20.937938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:20.937913    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:20:21.050253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050154    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-image-registry-private-configuration\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050215    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-trusted-ca\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050239    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-certificates\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050258    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-installation-pull-secrets\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050630 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050367    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-bound-sa-token\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050630 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050465    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050630 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050498    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89dqv\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-kube-api-access-89dqv\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050630 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050584    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c58df75d-dfcd-45eb-8778-e57ae06efc8d-ca-trust-extracted\") pod \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\" (UID: \"c58df75d-dfcd-45eb-8778-e57ae06efc8d\") "
Sep 16 02:20:21.050806 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050730    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:20:21.050806 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050741    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:20:21.050903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050840    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-trusted-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.050903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.050858    2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-certificates\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.053071 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.053038    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:20:21.053071 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.053066    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:20:21.053239 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.053074    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:20:21.053239 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.053167    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-kube-api-access-89dqv" (OuterVolumeSpecName: "kube-api-access-89dqv") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "kube-api-access-89dqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:20:21.053239 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.053167    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:20:21.058522 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.058497    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58df75d-dfcd-45eb-8778-e57ae06efc8d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c58df75d-dfcd-45eb-8778-e57ae06efc8d" (UID: "c58df75d-dfcd-45eb-8778-e57ae06efc8d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:20:21.152030 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.151993    2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c58df75d-dfcd-45eb-8778-e57ae06efc8d-ca-trust-extracted\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.152030 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.152022    2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-image-registry-private-configuration\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.152030 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.152033    2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c58df75d-dfcd-45eb-8778-e57ae06efc8d-installation-pull-secrets\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.152230 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.152043    2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-bound-sa-token\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.152230 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.152053    2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-registry-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.152230 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.152061    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dqv\" (UniqueName: \"kubernetes.io/projected/c58df75d-dfcd-45eb-8778-e57ae06efc8d-kube-api-access-89dqv\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:20:21.841367 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.841335    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4"
Sep 16 02:20:21.841891 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.841365    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6795b44c76-kkdw4" event={"ID":"c58df75d-dfcd-45eb-8778-e57ae06efc8d","Type":"ContainerDied","Data":"af196abc082726f5dc4e9768a36206d456c6fa22f9d645314d2e8574123f07c1"}
Sep 16 02:20:21.841891 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.841450    2556 scope.go:117] "RemoveContainer" containerID="f4f28cba1c394cd140ca90ebfef00b686a3b64a2f1bccc82684d0e389adbb4f9"
Sep 16 02:20:21.860105 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.860078    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6795b44c76-kkdw4"]
Sep 16 02:20:21.863864 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:21.863607    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6795b44c76-kkdw4"]
Sep 16 02:20:22.846271 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:22.846235    2556 generic.go:356] "Generic (PLEG): container finished" podID="dd5bdaad-9ef0-499f-a136-fefde5f6f7d2" containerID="dae260cf8a8b454ca207f1569aac64e1493bb22adb86a65f0e2321830b586d7e" exitCode=0
Sep 16 02:20:22.846740 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:22.846310    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8" event={"ID":"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2","Type":"ContainerDied","Data":"dae260cf8a8b454ca207f1569aac64e1493bb22adb86a65f0e2321830b586d7e"}
Sep 16 02:20:22.846740 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:22.846718    2556 scope.go:117] "RemoveContainer" containerID="dae260cf8a8b454ca207f1569aac64e1493bb22adb86a65f0e2321830b586d7e"
Sep 16 02:20:23.112775 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:23.112694    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c58df75d-dfcd-45eb-8778-e57ae06efc8d" path="/var/lib/kubelet/pods/c58df75d-dfcd-45eb-8778-e57ae06efc8d/volumes"
Sep 16 02:20:23.853028 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:23.852995    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-5gnj8" event={"ID":"dd5bdaad-9ef0-499f-a136-fefde5f6f7d2","Type":"ContainerStarted","Data":"234d27a9a337c339955d0dce23dab63b445137192765b40c2045aa73aa7cfb12"}
Sep 16 02:20:27.870151 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:27.870114    2556 generic.go:356] "Generic (PLEG): container finished" podID="36e5df43-16b6-4495-8396-ac2cc0c0bd0a" containerID="223f2d3d9606d2377daf3c07de551fde44c5269dd7d4ad11324ac1549a09ff4f" exitCode=0
Sep 16 02:20:27.870593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:27.870203    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl" event={"ID":"36e5df43-16b6-4495-8396-ac2cc0c0bd0a","Type":"ContainerDied","Data":"223f2d3d9606d2377daf3c07de551fde44c5269dd7d4ad11324ac1549a09ff4f"}
Sep 16 02:20:27.870593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:27.870559    2556 scope.go:117] "RemoveContainer" containerID="223f2d3d9606d2377daf3c07de551fde44c5269dd7d4ad11324ac1549a09ff4f"
Sep 16 02:20:28.876002 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:28.875964    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kkwcl" event={"ID":"36e5df43-16b6-4495-8396-ac2cc0c0bd0a","Type":"ContainerStarted","Data":"8d4d811e037c9f20ff8abc1218d30faf8c57627bd2bdfb403c259737b6eac63b"}
Sep 16 02:20:32.893005 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:32.892973    2556 generic.go:356] "Generic (PLEG): container finished" podID="177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42" containerID="80882e42589962afdee575fa2bdbdd6d3b68f2293fe9ab49203f570856318a03" exitCode=0
Sep 16 02:20:32.893366 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:32.893043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hs4wt" event={"ID":"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42","Type":"ContainerDied","Data":"80882e42589962afdee575fa2bdbdd6d3b68f2293fe9ab49203f570856318a03"}
Sep 16 02:20:32.893440 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:32.893365    2556 scope.go:117] "RemoveContainer" containerID="80882e42589962afdee575fa2bdbdd6d3b68f2293fe9ab49203f570856318a03"
Sep 16 02:20:33.898328 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:20:33.898291    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hs4wt" event={"ID":"177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42","Type":"ContainerStarted","Data":"1348ac3fa91304aee6e89f278215944341db0288e243e06d94e350e4fbf36d8d"}
Sep 16 02:23:07.516113 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.516077    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"]
Sep 16 02:23:07.516554 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.516360    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c58df75d-dfcd-45eb-8778-e57ae06efc8d" containerName="registry"
Sep 16 02:23:07.516554 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.516371    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c58df75d-dfcd-45eb-8778-e57ae06efc8d" containerName="registry"
Sep 16 02:23:07.516554 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.516458    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c58df75d-dfcd-45eb-8778-e57ae06efc8d" containerName="registry"
Sep 16 02:23:07.518387 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.518370    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.522538 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.522515    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 02:23:07.522700 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.522634    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 02:23:07.522834 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.522818    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 02:23:07.522907 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.522849    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 02:23:07.523095 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.523082    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tdbgx\""
Sep 16 02:23:07.534684 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.534662    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"]
Sep 16 02:23:07.562642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.562620    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64e8af4d-013a-411d-a3e7-f6d501b6d545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.562761 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.562649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e8af4d-013a-411d-a3e7-f6d501b6d545-tmp\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.562761 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.562669    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qp9\" (UniqueName: \"kubernetes.io/projected/64e8af4d-013a-411d-a3e7-f6d501b6d545-kube-api-access-z4qp9\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.562761 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.562687    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e8af4d-013a-411d-a3e7-f6d501b6d545-webhook-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.562904 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.562789    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64e8af4d-013a-411d-a3e7-f6d501b6d545-manifests\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.664200 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.664160    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64e8af4d-013a-411d-a3e7-f6d501b6d545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.664372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.664203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e8af4d-013a-411d-a3e7-f6d501b6d545-tmp\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.664372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.664225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qp9\" (UniqueName: \"kubernetes.io/projected/64e8af4d-013a-411d-a3e7-f6d501b6d545-kube-api-access-z4qp9\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.664372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.664243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e8af4d-013a-411d-a3e7-f6d501b6d545-webhook-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.664372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.664276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64e8af4d-013a-411d-a3e7-f6d501b6d545-manifests\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.664718 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.664697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64e8af4d-013a-411d-a3e7-f6d501b6d545-manifests\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.666396 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.666373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e8af4d-013a-411d-a3e7-f6d501b6d545-tmp\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.666754 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.666733    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64e8af4d-013a-411d-a3e7-f6d501b6d545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.666819 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.666766    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64e8af4d-013a-411d-a3e7-f6d501b6d545-webhook-cert\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.671329 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.671309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qp9\" (UniqueName: \"kubernetes.io/projected/64e8af4d-013a-411d-a3e7-f6d501b6d545-kube-api-access-z4qp9\") pod \"opendatahub-operator-controller-manager-599b75f66d-j2kgm\" (UID: \"64e8af4d-013a-411d-a3e7-f6d501b6d545\") " pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:07.828606 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:07.828524    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:08.103795 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:08.103771    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"]
Sep 16 02:23:08.106345 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:23:08.106316    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e8af4d_013a_411d_a3e7_f6d501b6d545.slice/crio-aec865d284cb6732404a81be07b1124401749a309da707e815ef6053dd9f431d WatchSource:0}: Error finding container aec865d284cb6732404a81be07b1124401749a309da707e815ef6053dd9f431d: Status 404 returned error can't find the container with id aec865d284cb6732404a81be07b1124401749a309da707e815ef6053dd9f431d
Sep 16 02:23:08.487728 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:08.487698    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm" event={"ID":"64e8af4d-013a-411d-a3e7-f6d501b6d545","Type":"ContainerStarted","Data":"aec865d284cb6732404a81be07b1124401749a309da707e815ef6053dd9f431d"}
Sep 16 02:23:11.502393 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:11.502355    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm" event={"ID":"64e8af4d-013a-411d-a3e7-f6d501b6d545","Type":"ContainerStarted","Data":"8e7442e9c9c633ec82e6124a6d27df0c2593cf9a5812be68283a3a2ee35e81ab"}
Sep 16 02:23:12.507960 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:12.507925    2556 generic.go:356] "Generic (PLEG): container finished" podID="64e8af4d-013a-411d-a3e7-f6d501b6d545" containerID="8e7442e9c9c633ec82e6124a6d27df0c2593cf9a5812be68283a3a2ee35e81ab" exitCode=0
Sep 16 02:23:12.508326 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:12.508002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm" event={"ID":"64e8af4d-013a-411d-a3e7-f6d501b6d545","Type":"ContainerDied","Data":"8e7442e9c9c633ec82e6124a6d27df0c2593cf9a5812be68283a3a2ee35e81ab"}
Sep 16 02:23:13.515543 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:13.515498    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm" event={"ID":"64e8af4d-013a-411d-a3e7-f6d501b6d545","Type":"ContainerStarted","Data":"7c08e0721760006549d2083901360681f5b1b41b6495bf2ea8df22cb8dfec40e"}
Sep 16 02:23:13.515918 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:13.515572    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:13.538761 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:13.538703    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm" podStartSLOduration=1.977869562 podStartE2EDuration="6.538681849s" totalImagesPullingTime="3.517773263s" totalInitContainerRuntime="1.043039024s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:13.535794514 +0000 UTC m=+292.965855902" watchObservedRunningTime="2026-09-16 02:23:13.538681849 +0000 UTC m=+292.968743235"
Sep 16 02:23:20.919085 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:20.919048    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:23:20.922295 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:20.922268    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:23:20.938768 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:20.938744    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 02:23:24.522809 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:24.522780    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-599b75f66d-j2kgm"
Sep 16 02:23:27.226283 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.226249    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"]
Sep 16 02:23:27.228481 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.228454    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.230645 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.230624    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 02:23:27.230764 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.230738    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 02:23:27.239409 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.239387    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"]
Sep 16 02:23:27.326688 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.326648    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswfm\" (UniqueName: \"kubernetes.io/projected/046a8542-6312-4e89-b4be-350a31a37f68-kube-api-access-xswfm\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.326836 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.326694    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a8542-6312-4e89-b4be-350a31a37f68-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.326836 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.326762    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046a8542-6312-4e89-b4be-350a31a37f68-tmp\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.428078 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.428048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046a8542-6312-4e89-b4be-350a31a37f68-tmp\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.428231 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.428113    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswfm\" (UniqueName: \"kubernetes.io/projected/046a8542-6312-4e89-b4be-350a31a37f68-kube-api-access-xswfm\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.428231 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.428134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a8542-6312-4e89-b4be-350a31a37f68-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.428231 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:23:27.428224    2556 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 02:23:27.428336 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:23:27.428275    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/046a8542-6312-4e89-b4be-350a31a37f68-tls-certs podName:046a8542-6312-4e89-b4be-350a31a37f68 nodeName:}" failed. No retries permitted until 2026-09-16 02:23:27.928260771 +0000 UTC m=+307.358322134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/046a8542-6312-4e89-b4be-350a31a37f68-tls-certs") pod "kube-auth-proxy-6f86d6bc5-9c7x6" (UID: "046a8542-6312-4e89-b4be-350a31a37f68") : secret "kube-auth-proxy-tls" not found
Sep 16 02:23:27.430259 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.430232    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046a8542-6312-4e89-b4be-350a31a37f68-tmp\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.437924 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.437902    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswfm\" (UniqueName: \"kubernetes.io/projected/046a8542-6312-4e89-b4be-350a31a37f68-kube-api-access-xswfm\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.931772 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.931732    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a8542-6312-4e89-b4be-350a31a37f68-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:27.934124 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:27.934102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a8542-6312-4e89-b4be-350a31a37f68-tls-certs\") pod \"kube-auth-proxy-6f86d6bc5-9c7x6\" (UID: \"046a8542-6312-4e89-b4be-350a31a37f68\") " pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:28.140340 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.140304    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"
Sep 16 02:23:28.415024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.414931    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6"]
Sep 16 02:23:28.417664 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:23:28.417630    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046a8542_6312_4e89_b4be_350a31a37f68.slice/crio-03306e44ff8d57c24b9b88cdfb25dcfea40bcbe9ddad011002d5d3a0fb24e7f9 WatchSource:0}: Error finding container 03306e44ff8d57c24b9b88cdfb25dcfea40bcbe9ddad011002d5d3a0fb24e7f9: Status 404 returned error can't find the container with id 03306e44ff8d57c24b9b88cdfb25dcfea40bcbe9ddad011002d5d3a0fb24e7f9
Sep 16 02:23:28.419724 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.419703    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:23:28.457931 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.457853    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"]
Sep 16 02:23:28.460637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.460616    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.462973 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.462952    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 02:23:28.463312 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.463174    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 02:23:28.463312 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.463192    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 02:23:28.463312 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.463223    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 02:23:28.463507 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.463492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q62wt\""
Sep 16 02:23:28.473044 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.473018    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"]
Sep 16 02:23:28.537178 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.537148    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6662p\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-kube-api-access-6662p\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.537294 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.537178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.537294 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.537198    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.537294 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.537239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.537294 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.537281    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1090dba3-1140-48e7-88b0-b636e06915ac-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.537424 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.537322    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.537424 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.537341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.578787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.578760    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6" event={"ID":"046a8542-6312-4e89-b4be-350a31a37f68","Type":"ContainerStarted","Data":"03306e44ff8d57c24b9b88cdfb25dcfea40bcbe9ddad011002d5d3a0fb24e7f9"}
Sep 16 02:23:28.637747 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.637719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1090dba3-1140-48e7-88b0-b636e06915ac-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.637869 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.637771    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.637869 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.637796    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.637869 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.637843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6662p\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-kube-api-access-6662p\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.637869 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.637860    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.638069 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.637883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.638069 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.637932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.638520 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.638474    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.640175 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.640151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.640272 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.640254    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1090dba3-1140-48e7-88b0-b636e06915ac-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.640346 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.640327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.640385 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.640354    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.645371 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.645350    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.646493 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.646411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6662p\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-kube-api-access-6662p\") pod \"istiod-openshift-gateway-7b7995d5cd-6pdc5\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:28.773328 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:28.773244    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:29.043902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:29.043875    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"]
Sep 16 02:23:29.045323 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:23:29.045303    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1090dba3_1140_48e7_88b0_b636e06915ac.slice/crio-0704b76b55bf99e0aedbdbd6fd5831e33c7d49c7ef0638f76bd67203cc5327e3 WatchSource:0}: Error finding container 0704b76b55bf99e0aedbdbd6fd5831e33c7d49c7ef0638f76bd67203cc5327e3: Status 404 returned error can't find the container with id 0704b76b55bf99e0aedbdbd6fd5831e33c7d49c7ef0638f76bd67203cc5327e3
Sep 16 02:23:29.586482 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:29.586424    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5" event={"ID":"1090dba3-1140-48e7-88b0-b636e06915ac","Type":"ContainerStarted","Data":"0704b76b55bf99e0aedbdbd6fd5831e33c7d49c7ef0638f76bd67203cc5327e3"}
Sep 16 02:23:32.526182 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:32.526146    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:23:32.526492 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:32.526228    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 02:23:32.604017 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:32.603988    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5" event={"ID":"1090dba3-1140-48e7-88b0-b636e06915ac","Type":"ContainerStarted","Data":"b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9"}
Sep 16 02:23:32.604147 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:32.604087    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:32.605780 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:32.605758    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6" event={"ID":"046a8542-6312-4e89-b4be-350a31a37f68","Type":"ContainerStarted","Data":"b733cd25bb6cca04c1ed8a7fb25e40083ce98c2fbd9d44bf2ac2871c1f581a60"}
Sep 16 02:23:32.625474 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:32.625386    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5" podStartSLOduration=1.146731886 podStartE2EDuration="4.625364138s" totalImagesPullingTime="3.478632252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:32.622539064 +0000 UTC m=+312.052600449" watchObservedRunningTime="2026-09-16 02:23:32.625364138 +0000 UTC m=+312.055425525"
Sep 16 02:23:32.637072 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:32.637019    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f86d6bc5-9c7x6" podStartSLOduration=1.529676732 podStartE2EDuration="5.637004008s" totalImagesPullingTime="4.107327276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:32.635572384 +0000 UTC m=+312.065633774" watchObservedRunningTime="2026-09-16 02:23:32.637004008 +0000 UTC m=+312.067065437"
Sep 16 02:23:33.611776 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:33.611749    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:47.128451 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.126925    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"]
Sep 16 02:23:47.131190 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.131164    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:47.136548 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.136505    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 02:23:47.137601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.137578    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zggv5\""
Sep 16 02:23:47.141743 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.141450    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"]
Sep 16 02:23:47.305104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.305062    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt48d\" (UniqueName: \"kubernetes.io/projected/a513ea27-fef0-43f9-96c8-8ca746e29a86-kube-api-access-bt48d\") pod \"llmisvc-controller-manager-777c6bd7d5-mbp22\" (UID: \"a513ea27-fef0-43f9-96c8-8ca746e29a86\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:47.305291 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.305214    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a513ea27-fef0-43f9-96c8-8ca746e29a86-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mbp22\" (UID: \"a513ea27-fef0-43f9-96c8-8ca746e29a86\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:47.405674 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.405589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a513ea27-fef0-43f9-96c8-8ca746e29a86-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mbp22\" (UID: \"a513ea27-fef0-43f9-96c8-8ca746e29a86\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:47.405674 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.405641    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt48d\" (UniqueName: \"kubernetes.io/projected/a513ea27-fef0-43f9-96c8-8ca746e29a86-kube-api-access-bt48d\") pod \"llmisvc-controller-manager-777c6bd7d5-mbp22\" (UID: \"a513ea27-fef0-43f9-96c8-8ca746e29a86\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:47.405896 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:23:47.405755    2556 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 02:23:47.405896 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:23:47.405829    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a513ea27-fef0-43f9-96c8-8ca746e29a86-cert podName:a513ea27-fef0-43f9-96c8-8ca746e29a86 nodeName:}" failed. No retries permitted until 2026-09-16 02:23:47.905809787 +0000 UTC m=+327.335871165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a513ea27-fef0-43f9-96c8-8ca746e29a86-cert") pod "llmisvc-controller-manager-777c6bd7d5-mbp22" (UID: "a513ea27-fef0-43f9-96c8-8ca746e29a86") : secret "llmisvc-webhook-server-cert" not found
Sep 16 02:23:47.414062 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.414036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt48d\" (UniqueName: \"kubernetes.io/projected/a513ea27-fef0-43f9-96c8-8ca746e29a86-kube-api-access-bt48d\") pod \"llmisvc-controller-manager-777c6bd7d5-mbp22\" (UID: \"a513ea27-fef0-43f9-96c8-8ca746e29a86\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:47.910338 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.910290    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a513ea27-fef0-43f9-96c8-8ca746e29a86-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mbp22\" (UID: \"a513ea27-fef0-43f9-96c8-8ca746e29a86\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:47.912985 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:47.912961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a513ea27-fef0-43f9-96c8-8ca746e29a86-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mbp22\" (UID: \"a513ea27-fef0-43f9-96c8-8ca746e29a86\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:48.047136 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:48.047105    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:48.322693 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:48.322662    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"]
Sep 16 02:23:48.325681 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:23:48.325647    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda513ea27_fef0_43f9_96c8_8ca746e29a86.slice/crio-5dd7f331c299441403f42bfd06b4b7f6c1e10dbc02acdf6bc559fe07bbef16b0 WatchSource:0}: Error finding container 5dd7f331c299441403f42bfd06b4b7f6c1e10dbc02acdf6bc559fe07bbef16b0: Status 404 returned error can't find the container with id 5dd7f331c299441403f42bfd06b4b7f6c1e10dbc02acdf6bc559fe07bbef16b0
Sep 16 02:23:48.676692 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:48.676648    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22" event={"ID":"a513ea27-fef0-43f9-96c8-8ca746e29a86","Type":"ContainerStarted","Data":"5dd7f331c299441403f42bfd06b4b7f6c1e10dbc02acdf6bc559fe07bbef16b0"}
Sep 16 02:23:48.920686 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:48.920656    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"]
Sep 16 02:23:48.920866 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:48.920838    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5" podUID="1090dba3-1140-48e7-88b0-b636e06915ac" containerName="discovery" containerID="cri-o://b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9" gracePeriod=30
Sep 16 02:23:49.220378 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.220345    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:49.321143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.321113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-istio-token\") pod \"1090dba3-1140-48e7-88b0-b636e06915ac\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") "
Sep 16 02:23:49.321300 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.321152    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1090dba3-1140-48e7-88b0-b636e06915ac-local-certs\") pod \"1090dba3-1140-48e7-88b0-b636e06915ac\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") "
Sep 16 02:23:49.321300 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.321195    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6662p\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-kube-api-access-6662p\") pod \"1090dba3-1140-48e7-88b0-b636e06915ac\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") "
Sep 16 02:23:49.321415 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.321315    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-cacerts\") pod \"1090dba3-1140-48e7-88b0-b636e06915ac\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") "
Sep 16 02:23:49.321415 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.321366    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-kubeconfig\") pod \"1090dba3-1140-48e7-88b0-b636e06915ac\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") "
Sep 16 02:23:49.321602 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.321441    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-ca-configmap\") pod \"1090dba3-1140-48e7-88b0-b636e06915ac\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") "
Sep 16 02:23:49.321602 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.321481    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-dns-cert\") pod \"1090dba3-1140-48e7-88b0-b636e06915ac\" (UID: \"1090dba3-1140-48e7-88b0-b636e06915ac\") "
Sep 16 02:23:49.322162 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.322096    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1090dba3-1140-48e7-88b0-b636e06915ac" (UID: "1090dba3-1140-48e7-88b0-b636e06915ac"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:23:49.323859 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.323816    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1090dba3-1140-48e7-88b0-b636e06915ac-local-certs" (OuterVolumeSpecName: "local-certs") pod "1090dba3-1140-48e7-88b0-b636e06915ac" (UID: "1090dba3-1140-48e7-88b0-b636e06915ac"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:23:49.323859 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.323817    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-cacerts" (OuterVolumeSpecName: "cacerts") pod "1090dba3-1140-48e7-88b0-b636e06915ac" (UID: "1090dba3-1140-48e7-88b0-b636e06915ac"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:23:49.323859 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.323844    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-kube-api-access-6662p" (OuterVolumeSpecName: "kube-api-access-6662p") pod "1090dba3-1140-48e7-88b0-b636e06915ac" (UID: "1090dba3-1140-48e7-88b0-b636e06915ac"). InnerVolumeSpecName "kube-api-access-6662p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:23:49.324238 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.323961    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "1090dba3-1140-48e7-88b0-b636e06915ac" (UID: "1090dba3-1140-48e7-88b0-b636e06915ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:23:49.324238 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.324123    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1090dba3-1140-48e7-88b0-b636e06915ac" (UID: "1090dba3-1140-48e7-88b0-b636e06915ac"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:23:49.324238 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.324131    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1090dba3-1140-48e7-88b0-b636e06915ac" (UID: "1090dba3-1140-48e7-88b0-b636e06915ac"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:23:49.422160 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.422129    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-ca-configmap\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:23:49.422160 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.422161    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-csr-dns-cert\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:23:49.422348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.422175    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-istio-token\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:23:49.422348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.422187    2556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1090dba3-1140-48e7-88b0-b636e06915ac-local-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:23:49.422348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.422199    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6662p\" (UniqueName: \"kubernetes.io/projected/1090dba3-1140-48e7-88b0-b636e06915ac-kube-api-access-6662p\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:23:49.422348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.422211    2556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-cacerts\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:23:49.422348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.422237    2556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1090dba3-1140-48e7-88b0-b636e06915ac-istio-kubeconfig\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:23:49.683893 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.683861    2556 generic.go:356] "Generic (PLEG): container finished" podID="1090dba3-1140-48e7-88b0-b636e06915ac" containerID="b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9" exitCode=0
Sep 16 02:23:49.684063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.683935    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"
Sep 16 02:23:49.684063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.683948    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5" event={"ID":"1090dba3-1140-48e7-88b0-b636e06915ac","Type":"ContainerDied","Data":"b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9"}
Sep 16 02:23:49.684063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.684026    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5" event={"ID":"1090dba3-1140-48e7-88b0-b636e06915ac","Type":"ContainerDied","Data":"0704b76b55bf99e0aedbdbd6fd5831e33c7d49c7ef0638f76bd67203cc5327e3"}
Sep 16 02:23:49.684063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.684055    2556 scope.go:117] "RemoveContainer" containerID="b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9"
Sep 16 02:23:49.693550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.693442    2556 scope.go:117] "RemoveContainer" containerID="b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9"
Sep 16 02:23:49.693928 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:23:49.693752    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9\": container with ID starting with b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9 not found: ID does not exist" containerID="b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9"
Sep 16 02:23:49.693928 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.693785    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9"} err="failed to get container status \"b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9\": rpc error: code = NotFound desc = could not find container \"b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9\": container with ID starting with b25db8ade09d2b3a13628c1aac1dc58cf491e4465895512c68bb3de48fff4dd9 not found: ID does not exist"
Sep 16 02:23:49.708045 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.708015    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"]
Sep 16 02:23:49.714874 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:49.714850    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6pdc5"]
Sep 16 02:23:51.112657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:51.112635    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1090dba3-1140-48e7-88b0-b636e06915ac" path="/var/lib/kubelet/pods/1090dba3-1140-48e7-88b0-b636e06915ac/volumes"
Sep 16 02:23:51.697367 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:51.697329    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22" event={"ID":"a513ea27-fef0-43f9-96c8-8ca746e29a86","Type":"ContainerStarted","Data":"df051ca9c9879ea2ac3e5b15b4a0136e097d5ce2ac2e721c1dd225107c48fb6f"}
Sep 16 02:23:51.697564 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:51.697498    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:23:51.713840 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:23:51.713796    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22" podStartSLOduration=1.97414412 podStartE2EDuration="4.71378313s" totalImagesPullingTime="2.73963901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:23:51.71257278 +0000 UTC m=+331.142634166" watchObservedRunningTime="2026-09-16 02:23:51.71378313 +0000 UTC m=+331.143844516"
Sep 16 02:24:22.705196 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:22.705156    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mbp22"
Sep 16 02:24:54.847096 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.847060    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p4x6c"]
Sep 16 02:24:54.847626 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.847370    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1090dba3-1140-48e7-88b0-b636e06915ac" containerName="discovery"
Sep 16 02:24:54.847626 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.847382    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1090dba3-1140-48e7-88b0-b636e06915ac" containerName="discovery"
Sep 16 02:24:54.847626 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.847465    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1090dba3-1140-48e7-88b0-b636e06915ac" containerName="discovery"
Sep 16 02:24:54.850517 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.850498    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c"
Sep 16 02:24:54.852943 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.852917    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x66mp\""
Sep 16 02:24:54.853090 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.852917    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 02:24:54.853090 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.852960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 02:24:54.861694 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.861664    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p4x6c"]
Sep 16 02:24:54.957156 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:54.957102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9vv\" (UniqueName: \"kubernetes.io/projected/ea5dc572-039c-4628-9d52-442525df6346-kube-api-access-zf9vv\") pod \"authorino-operator-76b478bd78-p4x6c\" (UID: \"ea5dc572-039c-4628-9d52-442525df6346\") " pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c"
Sep 16 02:24:55.058386 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:55.058342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9vv\" (UniqueName: \"kubernetes.io/projected/ea5dc572-039c-4628-9d52-442525df6346-kube-api-access-zf9vv\") pod \"authorino-operator-76b478bd78-p4x6c\" (UID: \"ea5dc572-039c-4628-9d52-442525df6346\") " pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c"
Sep 16 02:24:55.066047 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:55.066011    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9vv\" (UniqueName: \"kubernetes.io/projected/ea5dc572-039c-4628-9d52-442525df6346-kube-api-access-zf9vv\") pod \"authorino-operator-76b478bd78-p4x6c\" (UID: \"ea5dc572-039c-4628-9d52-442525df6346\") " pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c"
Sep 16 02:24:55.161412 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:55.161319    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c"
Sep 16 02:24:55.435897 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:55.435864    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p4x6c"]
Sep 16 02:24:55.439376 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:24:55.439347    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5dc572_039c_4628_9d52_442525df6346.slice/crio-d3289339972554b926e990f6119ef5ca6088498960ddd153161aed64fcc276ef WatchSource:0}: Error finding container d3289339972554b926e990f6119ef5ca6088498960ddd153161aed64fcc276ef: Status 404 returned error can't find the container with id d3289339972554b926e990f6119ef5ca6088498960ddd153161aed64fcc276ef
Sep 16 02:24:55.982509 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:55.982468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c" event={"ID":"ea5dc572-039c-4628-9d52-442525df6346","Type":"ContainerStarted","Data":"d3289339972554b926e990f6119ef5ca6088498960ddd153161aed64fcc276ef"}
Sep 16 02:24:57.028128 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.028087    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"]
Sep 16 02:24:57.034607 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.034582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:24:57.036968 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.036946    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bvvft\""
Sep 16 02:24:57.043698 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.043671    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"]
Sep 16 02:24:57.073243 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.073210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blk9g\" (UniqueName: \"kubernetes.io/projected/93a3c3a4-b89c-4cd4-ae9e-6ad152b53318-kube-api-access-blk9g\") pod \"limitador-operator-controller-manager-74495c69f8-wm658\" (UID: \"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:24:57.174593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.174552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blk9g\" (UniqueName: \"kubernetes.io/projected/93a3c3a4-b89c-4cd4-ae9e-6ad152b53318-kube-api-access-blk9g\") pod \"limitador-operator-controller-manager-74495c69f8-wm658\" (UID: \"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:24:57.187914 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.187880    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blk9g\" (UniqueName: \"kubernetes.io/projected/93a3c3a4-b89c-4cd4-ae9e-6ad152b53318-kube-api-access-blk9g\") pod \"limitador-operator-controller-manager-74495c69f8-wm658\" (UID: \"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:24:57.347121 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.347038    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:24:57.993740 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.993704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c" event={"ID":"ea5dc572-039c-4628-9d52-442525df6346","Type":"ContainerStarted","Data":"444b8335cfad82811b2e1ebf7332dc3960f7817ef7c87b8595144ed9f8df7e5d"}
Sep 16 02:24:57.993914 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:57.993892    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c"
Sep 16 02:24:58.016110 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:58.016062    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c" podStartSLOduration=1.573855553 podStartE2EDuration="4.016047486s" totalImagesPullingTime="2.442191933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:24:58.014782185 +0000 UTC m=+397.444843570" watchObservedRunningTime="2026-09-16 02:24:58.016047486 +0000 UTC m=+397.446108872"
Sep 16 02:24:58.084656 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:58.084631    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"]
Sep 16 02:24:58.087200 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:24:58.087174    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a3c3a4_b89c_4cd4_ae9e_6ad152b53318.slice/crio-852fe342b68f116df094239b15f2a5279b659d40856517efde3d6bce9526b66e WatchSource:0}: Error finding container 852fe342b68f116df094239b15f2a5279b659d40856517efde3d6bce9526b66e: Status 404 returned error can't find the container with id 852fe342b68f116df094239b15f2a5279b659d40856517efde3d6bce9526b66e
Sep 16 02:24:59.000809 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.000773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658" event={"ID":"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318","Type":"ContainerStarted","Data":"852fe342b68f116df094239b15f2a5279b659d40856517efde3d6bce9526b66e"}
Sep 16 02:24:59.349016 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.348923    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"]
Sep 16 02:24:59.352679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.352628    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"
Sep 16 02:24:59.355229 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.355204    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5bchj\""
Sep 16 02:24:59.355341 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.355264    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 02:24:59.364187 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.364147    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"]
Sep 16 02:24:59.391366 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.391326    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmz8\" (UniqueName: \"kubernetes.io/projected/3bad8a8b-9834-47c6-ac7c-6c5479a9e56d-kube-api-access-4nmz8\") pod \"dns-operator-controller-manager-7b84cd7c9c-gxhc5\" (UID: \"3bad8a8b-9834-47c6-ac7c-6c5479a9e56d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"
Sep 16 02:24:59.492553 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.492510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmz8\" (UniqueName: \"kubernetes.io/projected/3bad8a8b-9834-47c6-ac7c-6c5479a9e56d-kube-api-access-4nmz8\") pod \"dns-operator-controller-manager-7b84cd7c9c-gxhc5\" (UID: \"3bad8a8b-9834-47c6-ac7c-6c5479a9e56d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"
Sep 16 02:24:59.512346 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.512306    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmz8\" (UniqueName: \"kubernetes.io/projected/3bad8a8b-9834-47c6-ac7c-6c5479a9e56d-kube-api-access-4nmz8\") pod \"dns-operator-controller-manager-7b84cd7c9c-gxhc5\" (UID: \"3bad8a8b-9834-47c6-ac7c-6c5479a9e56d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"
Sep 16 02:24:59.665421 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.665393    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"
Sep 16 02:24:59.968061 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:24:59.968035    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"]
Sep 16 02:24:59.970470 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:24:59.970444    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bad8a8b_9834_47c6_ac7c_6c5479a9e56d.slice/crio-e006026e78beb73062757a519de04f7982905936608edc07ce542ce2e4d241f2 WatchSource:0}: Error finding container e006026e78beb73062757a519de04f7982905936608edc07ce542ce2e4d241f2: Status 404 returned error can't find the container with id e006026e78beb73062757a519de04f7982905936608edc07ce542ce2e4d241f2
Sep 16 02:25:00.006722 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:00.006686    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5" event={"ID":"3bad8a8b-9834-47c6-ac7c-6c5479a9e56d","Type":"ContainerStarted","Data":"e006026e78beb73062757a519de04f7982905936608edc07ce542ce2e4d241f2"}
Sep 16 02:25:00.008393 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:00.008326    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658" event={"ID":"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318","Type":"ContainerStarted","Data":"77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c"}
Sep 16 02:25:00.008534 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:00.008464    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:00.031125 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:00.031076    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658" podStartSLOduration=1.53898278 podStartE2EDuration="3.031058567s" totalImagesPullingTime="1.492075787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:00.030240012 +0000 UTC m=+399.460301411" watchObservedRunningTime="2026-09-16 02:25:00.031058567 +0000 UTC m=+399.461119953"
Sep 16 02:25:02.022652 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:02.022617    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5" event={"ID":"3bad8a8b-9834-47c6-ac7c-6c5479a9e56d","Type":"ContainerStarted","Data":"75cd45fe2fa4631d76ae4b25655bbec49792c035ea34f41945c5ef89d7bf60be"}
Sep 16 02:25:02.023040 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:02.022680    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"
Sep 16 02:25:02.041107 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:02.041056    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5" podStartSLOduration=1.48926662 podStartE2EDuration="3.041042129s" totalImagesPullingTime="1.551775509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:02.038507875 +0000 UTC m=+401.468569262" watchObservedRunningTime="2026-09-16 02:25:02.041042129 +0000 UTC m=+401.471103515"
Sep 16 02:25:03.462637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.462599    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"]
Sep 16 02:25:03.466228 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.466211    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:03.468586 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.468564    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vbcj6\""
Sep 16 02:25:03.482466 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.482420    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"]
Sep 16 02:25:03.523971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.523934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwm9m\" (UniqueName: \"kubernetes.io/projected/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-kube-api-access-dwm9m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:03.523971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.523974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:03.625133 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.625092    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwm9m\" (UniqueName: \"kubernetes.io/projected/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-kube-api-access-dwm9m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:03.625133 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.625133    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:03.625518 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.625497    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:03.634344 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.634312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwm9m\" (UniqueName: \"kubernetes.io/projected/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-kube-api-access-dwm9m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:03.775980 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:03.775884    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:04.057454 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:04.057412    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"]
Sep 16 02:25:04.059375 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:25:04.059347    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15d1070_f3aa_43ff_8150_f5b4ab7fbec2.slice/crio-af0caa31baf6b4b7a3456d4108a2c885cfbe566dfea320e0416aa1ae85aca0bb WatchSource:0}: Error finding container af0caa31baf6b4b7a3456d4108a2c885cfbe566dfea320e0416aa1ae85aca0bb: Status 404 returned error can't find the container with id af0caa31baf6b4b7a3456d4108a2c885cfbe566dfea320e0416aa1ae85aca0bb
Sep 16 02:25:05.037564 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:05.037525    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6" event={"ID":"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2","Type":"ContainerStarted","Data":"af0caa31baf6b4b7a3456d4108a2c885cfbe566dfea320e0416aa1ae85aca0bb"}
Sep 16 02:25:09.003555 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:09.003522    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-p4x6c"
Sep 16 02:25:09.060756 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:09.060721    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6" event={"ID":"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2","Type":"ContainerStarted","Data":"42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675"}
Sep 16 02:25:09.060941 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:09.060922    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:09.080609 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:09.080545    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6" podStartSLOduration=1.573158026 podStartE2EDuration="6.080524977s" totalImagesPullingTime="4.507366951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:09.078327563 +0000 UTC m=+408.508388949" watchObservedRunningTime="2026-09-16 02:25:09.080524977 +0000 UTC m=+408.510586364"
Sep 16 02:25:11.018378 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.018349    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:11.216135 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.216103    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"]
Sep 16 02:25:11.219530 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.219509    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.221900 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.221877    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qn5l4\""
Sep 16 02:25:11.222077 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.221876    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 02:25:11.222146 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.221895    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 02:25:11.232262 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.232233    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"]
Sep 16 02:25:11.296467 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.296349    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7430254-3fe5-44c1-b21f-67a26c46bff8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.296467 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.296400    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7430254-3fe5-44c1-b21f-67a26c46bff8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.296715 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.296488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhp2\" (UniqueName: \"kubernetes.io/projected/d7430254-3fe5-44c1-b21f-67a26c46bff8-kube-api-access-mvhp2\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.397348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.397312    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7430254-3fe5-44c1-b21f-67a26c46bff8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.397545 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.397360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7430254-3fe5-44c1-b21f-67a26c46bff8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.397545 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.397401    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhp2\" (UniqueName: \"kubernetes.io/projected/d7430254-3fe5-44c1-b21f-67a26c46bff8-kube-api-access-mvhp2\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.397545 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:11.397528    2556 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 02:25:11.397721 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:11.397603    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7430254-3fe5-44c1-b21f-67a26c46bff8-plugin-serving-cert podName:d7430254-3fe5-44c1-b21f-67a26c46bff8 nodeName:}" failed. No retries permitted until 2026-09-16 02:25:11.89758399 +0000 UTC m=+411.327645354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d7430254-3fe5-44c1-b21f-67a26c46bff8-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-797p4" (UID: "d7430254-3fe5-44c1-b21f-67a26c46bff8") : secret "plugin-serving-cert" not found
Sep 16 02:25:11.397981 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.397961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7430254-3fe5-44c1-b21f-67a26c46bff8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.405679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.405653    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhp2\" (UniqueName: \"kubernetes.io/projected/d7430254-3fe5-44c1-b21f-67a26c46bff8-kube-api-access-mvhp2\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.901040 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.900999    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7430254-3fe5-44c1-b21f-67a26c46bff8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:11.903414 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:11.903390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7430254-3fe5-44c1-b21f-67a26c46bff8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-797p4\" (UID: \"d7430254-3fe5-44c1-b21f-67a26c46bff8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:12.129220 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:12.129168    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"
Sep 16 02:25:12.397993 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:12.397952    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4"]
Sep 16 02:25:12.401058 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:25:12.401024    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7430254_3fe5_44c1_b21f_67a26c46bff8.slice/crio-f28726d54eefe392704588f596e7dd5dc2d899a2f36985f5bb879e1eb70b389f WatchSource:0}: Error finding container f28726d54eefe392704588f596e7dd5dc2d899a2f36985f5bb879e1eb70b389f: Status 404 returned error can't find the container with id f28726d54eefe392704588f596e7dd5dc2d899a2f36985f5bb879e1eb70b389f
Sep 16 02:25:13.029411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:13.029381    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gxhc5"
Sep 16 02:25:13.081421 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:13.081387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4" event={"ID":"d7430254-3fe5-44c1-b21f-67a26c46bff8","Type":"ContainerStarted","Data":"f28726d54eefe392704588f596e7dd5dc2d899a2f36985f5bb879e1eb70b389f"}
Sep 16 02:25:17.102556 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:17.102521    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4" event={"ID":"d7430254-3fe5-44c1-b21f-67a26c46bff8","Type":"ContainerStarted","Data":"b09a11a007493c36871588b964b35ab5f66b5a40f769eba861ca3c4037a7fb11"}
Sep 16 02:25:17.117976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:17.117917    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-797p4" podStartSLOduration=1.515537723 podStartE2EDuration="6.117902575s" totalImagesPullingTime="4.602364852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:17.116201729 +0000 UTC m=+416.546263095" watchObservedRunningTime="2026-09-16 02:25:17.117902575 +0000 UTC m=+416.547963961"
Sep 16 02:25:20.068372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:20.068340    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.334994 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.334953    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"]
Sep 16 02:25:22.335413 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.335156    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" containerName="manager" containerID="cri-o://42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675" gracePeriod=2
Sep 16 02:25:22.344908 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.344875    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"]
Sep 16 02:25:22.345828 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.345795    2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\": pods \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.352600 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.352557    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"]
Sep 16 02:25:22.352851 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.352796    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" containerName="manager" containerID="cri-o://77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c" gracePeriod=2
Sep 16 02:25:22.356164 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.355915    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"]
Sep 16 02:25:22.356505 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.356486    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" containerName="manager"
Sep 16 02:25:22.356674 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.356661    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" containerName="manager"
Sep 16 02:25:22.356890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.356857    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" containerName="manager"
Sep 16 02:25:22.360915 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.360892    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.365961 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.365935    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"]
Sep 16 02:25:22.375665 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.375624    2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-wm658\": pods \"limitador-operator-controller-manager-74495c69f8-wm658\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:22.378153 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.378109    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.379808 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.379775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"]
Sep 16 02:25:22.380000 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.379971    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.381697 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.381667    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wm658\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:22.381812 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.381756    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"]
Sep 16 02:25:22.382395 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.382164    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" containerName="manager"
Sep 16 02:25:22.382395 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.382185    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" containerName="manager"
Sep 16 02:25:22.382395 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.382301    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" containerName="manager"
Sep 16 02:25:22.385324 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.385305    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"
Sep 16 02:25:22.402554 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.402518    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"]
Sep 16 02:25:22.428840 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.428802    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.430695 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.430657    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wm658\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:22.493821 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.493783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7drd\" (UniqueName: \"kubernetes.io/projected/30aaa2da-a0fe-45dd-99d0-f47c36b89760-kube-api-access-v7drd\") pod \"limitador-operator-controller-manager-74495c69f8-5r7qc\" (UID: \"30aaa2da-a0fe-45dd-99d0-f47c36b89760\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"
Sep 16 02:25:22.493962 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.493835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938ae59d-320d-4b66-aa2d-aff848c5c972-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-njrmz\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.493962 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.493867    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpgc\" (UniqueName: \"kubernetes.io/projected/938ae59d-320d-4b66-aa2d-aff848c5c972-kube-api-access-kvpgc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-njrmz\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.594654 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.594558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7drd\" (UniqueName: \"kubernetes.io/projected/30aaa2da-a0fe-45dd-99d0-f47c36b89760-kube-api-access-v7drd\") pod \"limitador-operator-controller-manager-74495c69f8-5r7qc\" (UID: \"30aaa2da-a0fe-45dd-99d0-f47c36b89760\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"
Sep 16 02:25:22.594654 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.594606    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938ae59d-320d-4b66-aa2d-aff848c5c972-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-njrmz\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.594654 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.594636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpgc\" (UniqueName: \"kubernetes.io/projected/938ae59d-320d-4b66-aa2d-aff848c5c972-kube-api-access-kvpgc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-njrmz\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.595006 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.594984    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938ae59d-320d-4b66-aa2d-aff848c5c972-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-njrmz\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.602528 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.602502    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpgc\" (UniqueName: \"kubernetes.io/projected/938ae59d-320d-4b66-aa2d-aff848c5c972-kube-api-access-kvpgc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-njrmz\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.602646 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.602624    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7drd\" (UniqueName: \"kubernetes.io/projected/30aaa2da-a0fe-45dd-99d0-f47c36b89760-kube-api-access-v7drd\") pod \"limitador-operator-controller-manager-74495c69f8-5r7qc\" (UID: \"30aaa2da-a0fe-45dd-99d0-f47c36b89760\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"
Sep 16 02:25:22.625342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.625315    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:22.627438 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.627380    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wm658\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:22.627538 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.627514    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.629012 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.628989    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.630648 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.630626    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xdjw6\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:22.632321 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:22.632298    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-wm658\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:22.695380 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.695337    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blk9g\" (UniqueName: \"kubernetes.io/projected/93a3c3a4-b89c-4cd4-ae9e-6ad152b53318-kube-api-access-blk9g\") pod \"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318\" (UID: \"93a3c3a4-b89c-4cd4-ae9e-6ad152b53318\") "
Sep 16 02:25:22.695591 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.695530    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwm9m\" (UniqueName: \"kubernetes.io/projected/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-kube-api-access-dwm9m\") pod \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") "
Sep 16 02:25:22.695591 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.695564    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-extensions-socket-volume\") pod \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\" (UID: \"c15d1070-f3aa-43ff-8150-f5b4ab7fbec2\") "
Sep 16 02:25:22.695976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.695948    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" (UID: "c15d1070-f3aa-43ff-8150-f5b4ab7fbec2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:25:22.697713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.697683    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-kube-api-access-dwm9m" (OuterVolumeSpecName: "kube-api-access-dwm9m") pod "c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" (UID: "c15d1070-f3aa-43ff-8150-f5b4ab7fbec2"). InnerVolumeSpecName "kube-api-access-dwm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:25:22.697795 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.697715    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a3c3a4-b89c-4cd4-ae9e-6ad152b53318-kube-api-access-blk9g" (OuterVolumeSpecName: "kube-api-access-blk9g") pod "93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" (UID: "93a3c3a4-b89c-4cd4-ae9e-6ad152b53318"). InnerVolumeSpecName "kube-api-access-blk9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:25:22.786094 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.786053    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:22.794862 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.794646    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"
Sep 16 02:25:22.797050 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.797024    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwm9m\" (UniqueName: \"kubernetes.io/projected/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-kube-api-access-dwm9m\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:25:22.797050 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.797048    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2-extensions-socket-volume\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:25:22.797282 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:22.797060    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blk9g\" (UniqueName: \"kubernetes.io/projected/93a3c3a4-b89c-4cd4-ae9e-6ad152b53318-kube-api-access-blk9g\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:25:23.073381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.073326    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"]
Sep 16 02:25:23.075330 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:25:23.075302    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938ae59d_320d_4b66_aa2d_aff848c5c972.slice/crio-c140f07154370fdb818de9d9323c6f4ced7cd9a7f4b15e80f6b3fc2d028b764c WatchSource:0}: Error finding container c140f07154370fdb818de9d9323c6f4ced7cd9a7f4b15e80f6b3fc2d028b764c: Status 404 returned error can't find the container with id c140f07154370fdb818de9d9323c6f4ced7cd9a7f4b15e80f6b3fc2d028b764c
Sep 16 02:25:23.076162 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.076141    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"]
Sep 16 02:25:23.078512 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:25:23.078486    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30aaa2da_a0fe_45dd_99d0_f47c36b89760.slice/crio-e8d6d4c0b355519888da6e8772c9cbf591c704edd5957487073f410d15a68ee5 WatchSource:0}: Error finding container e8d6d4c0b355519888da6e8772c9cbf591c704edd5957487073f410d15a68ee5: Status 404 returned error can't find the container with id e8d6d4c0b355519888da6e8772c9cbf591c704edd5957487073f410d15a68ee5
Sep 16 02:25:23.119969 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.117677    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" path="/var/lib/kubelet/pods/93a3c3a4-b89c-4cd4-ae9e-6ad152b53318/volumes"
Sep 16 02:25:23.119969 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.118207    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" path="/var/lib/kubelet/pods/c15d1070-f3aa-43ff-8150-f5b4ab7fbec2/volumes"
Sep 16 02:25:23.133384 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.133352    2556 generic.go:356] "Generic (PLEG): container finished" podID="c15d1070-f3aa-43ff-8150-f5b4ab7fbec2" containerID="42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675" exitCode=0
Sep 16 02:25:23.133527 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.133406    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xdjw6"
Sep 16 02:25:23.133527 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.133454    2556 scope.go:117] "RemoveContainer" containerID="42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675"
Sep 16 02:25:23.135130 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.135090    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc" event={"ID":"30aaa2da-a0fe-45dd-99d0-f47c36b89760","Type":"ContainerStarted","Data":"e8d6d4c0b355519888da6e8772c9cbf591c704edd5957487073f410d15a68ee5"}
Sep 16 02:25:23.138466 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.138333    2556 generic.go:356] "Generic (PLEG): container finished" podID="93a3c3a4-b89c-4cd4-ae9e-6ad152b53318" containerID="77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c" exitCode=0
Sep 16 02:25:23.138571 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.138476    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wm658"
Sep 16 02:25:23.141863 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.141839    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz" event={"ID":"938ae59d-320d-4b66-aa2d-aff848c5c972","Type":"ContainerStarted","Data":"c140f07154370fdb818de9d9323c6f4ced7cd9a7f4b15e80f6b3fc2d028b764c"}
Sep 16 02:25:23.146048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.145952    2556 scope.go:117] "RemoveContainer" containerID="42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675"
Sep 16 02:25:23.146286 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:23.146261    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675\": container with ID starting with 42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675 not found: ID does not exist" containerID="42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675"
Sep 16 02:25:23.146376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.146296    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675"} err="failed to get container status \"42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675\": rpc error: code = NotFound desc = could not find container \"42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675\": container with ID starting with 42e4617be1c848ab4845c295788289c9809e853553e359bb8258ebbe4da94675 not found: ID does not exist"
Sep 16 02:25:23.146376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.146320    2556 scope.go:117] "RemoveContainer" containerID="77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c"
Sep 16 02:25:23.154127 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.154105    2556 scope.go:117] "RemoveContainer" containerID="77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c"
Sep 16 02:25:23.154394 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:23.154367    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c\": container with ID starting with 77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c not found: ID does not exist" containerID="77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c"
Sep 16 02:25:23.154480 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:23.154406    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c"} err="failed to get container status \"77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c\": rpc error: code = NotFound desc = could not find container \"77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c\": container with ID starting with 77bba13ff44471b4b3aa49894bbda0b5c56a2c0684b1fbc9ecf0c6e3fc7f867c not found: ID does not exist"
Sep 16 02:25:24.150809 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:24.150774    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc" event={"ID":"30aaa2da-a0fe-45dd-99d0-f47c36b89760","Type":"ContainerStarted","Data":"16dc1c87883faa3dcb182f85e6af2f376968d01d889ea274ec078d9730c07726"}
Sep 16 02:25:24.151269 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:24.150894    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"
Sep 16 02:25:24.153624 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:24.153586    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz" event={"ID":"938ae59d-320d-4b66-aa2d-aff848c5c972","Type":"ContainerStarted","Data":"39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553"}
Sep 16 02:25:24.153771 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:24.153750    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:24.168140 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:24.168086    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc" podStartSLOduration=2.168068421 podStartE2EDuration="2.168068421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:24.166544907 +0000 UTC m=+423.596606306" watchObservedRunningTime="2026-09-16 02:25:24.168068421 +0000 UTC m=+423.598129810"
Sep 16 02:25:24.184842 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:24.184799    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz" podStartSLOduration=2.18478432 podStartE2EDuration="2.18478432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:24.183155493 +0000 UTC m=+423.613216881" watchObservedRunningTime="2026-09-16 02:25:24.18478432 +0000 UTC m=+423.614845706"
Sep 16 02:25:35.160768 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:35.160734    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5r7qc"
Sep 16 02:25:35.161306 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:35.160938    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:36.940022 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:36.939977    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"]
Sep 16 02:25:36.940508 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:36.940290    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz" podUID="938ae59d-320d-4b66-aa2d-aff848c5c972" containerName="manager" containerID="cri-o://39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553" gracePeriod=10
Sep 16 02:25:37.148343 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.148309    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"]
Sep 16 02:25:37.150583 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.150567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.163935 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.163908    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"]
Sep 16 02:25:37.185155 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.185135    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:37.218708 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.218629    2556 generic.go:356] "Generic (PLEG): container finished" podID="938ae59d-320d-4b66-aa2d-aff848c5c972" containerID="39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553" exitCode=0
Sep 16 02:25:37.218708 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.218677    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"
Sep 16 02:25:37.218874 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.218722    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz" event={"ID":"938ae59d-320d-4b66-aa2d-aff848c5c972","Type":"ContainerDied","Data":"39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553"}
Sep 16 02:25:37.218874 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.218778    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz" event={"ID":"938ae59d-320d-4b66-aa2d-aff848c5c972","Type":"ContainerDied","Data":"c140f07154370fdb818de9d9323c6f4ced7cd9a7f4b15e80f6b3fc2d028b764c"}
Sep 16 02:25:37.218874 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.218804    2556 scope.go:117] "RemoveContainer" containerID="39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553"
Sep 16 02:25:37.224188 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.224162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fabda73-1c04-4032-8a25-2ab7351f572e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xhn4j\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.224273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.224243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jks\" (UniqueName: \"kubernetes.io/projected/0fabda73-1c04-4032-8a25-2ab7351f572e-kube-api-access-s7jks\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xhn4j\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.226756 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.226728    2556 scope.go:117] "RemoveContainer" containerID="39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553"
Sep 16 02:25:37.226999 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:25:37.226981    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553\": container with ID starting with 39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553 not found: ID does not exist" containerID="39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553"
Sep 16 02:25:37.227043 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.227008    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553"} err="failed to get container status \"39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553\": rpc error: code = NotFound desc = could not find container \"39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553\": container with ID starting with 39b9e77763b221530ba4f5634b303d90966ea715d138eddc4497a6ca8b5ab553 not found: ID does not exist"
Sep 16 02:25:37.324716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.324677    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938ae59d-320d-4b66-aa2d-aff848c5c972-extensions-socket-volume\") pod \"938ae59d-320d-4b66-aa2d-aff848c5c972\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") "
Sep 16 02:25:37.324716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.324713    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvpgc\" (UniqueName: \"kubernetes.io/projected/938ae59d-320d-4b66-aa2d-aff848c5c972-kube-api-access-kvpgc\") pod \"938ae59d-320d-4b66-aa2d-aff848c5c972\" (UID: \"938ae59d-320d-4b66-aa2d-aff848c5c972\") "
Sep 16 02:25:37.324969 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.324806    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fabda73-1c04-4032-8a25-2ab7351f572e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xhn4j\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.324969 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.324913    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jks\" (UniqueName: \"kubernetes.io/projected/0fabda73-1c04-4032-8a25-2ab7351f572e-kube-api-access-s7jks\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xhn4j\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.325153 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.325134    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938ae59d-320d-4b66-aa2d-aff848c5c972-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "938ae59d-320d-4b66-aa2d-aff848c5c972" (UID: "938ae59d-320d-4b66-aa2d-aff848c5c972"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:25:37.325262 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.325238    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fabda73-1c04-4032-8a25-2ab7351f572e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xhn4j\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.326831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.326808    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938ae59d-320d-4b66-aa2d-aff848c5c972-kube-api-access-kvpgc" (OuterVolumeSpecName: "kube-api-access-kvpgc") pod "938ae59d-320d-4b66-aa2d-aff848c5c972" (UID: "938ae59d-320d-4b66-aa2d-aff848c5c972"). InnerVolumeSpecName "kube-api-access-kvpgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:25:37.334618 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.334597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jks\" (UniqueName: \"kubernetes.io/projected/0fabda73-1c04-4032-8a25-2ab7351f572e-kube-api-access-s7jks\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xhn4j\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.425951 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.425913    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/938ae59d-320d-4b66-aa2d-aff848c5c972-extensions-socket-volume\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:25:37.425951 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.425952    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvpgc\" (UniqueName: \"kubernetes.io/projected/938ae59d-320d-4b66-aa2d-aff848c5c972-kube-api-access-kvpgc\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:25:37.460935 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.460902    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:37.544385 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.544353    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"]
Sep 16 02:25:37.548866 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.548836    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-njrmz"]
Sep 16 02:25:37.732746 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:37.732721    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"]
Sep 16 02:25:37.735119 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:25:37.735096    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fabda73_1c04_4032_8a25_2ab7351f572e.slice/crio-e4091772abdb5b5db12599641f1545a89d809373bf55229cdb15be8ef500ab07 WatchSource:0}: Error finding container e4091772abdb5b5db12599641f1545a89d809373bf55229cdb15be8ef500ab07: Status 404 returned error can't find the container with id e4091772abdb5b5db12599641f1545a89d809373bf55229cdb15be8ef500ab07
Sep 16 02:25:38.225443 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:38.225395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j" event={"ID":"0fabda73-1c04-4032-8a25-2ab7351f572e","Type":"ContainerStarted","Data":"86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c"}
Sep 16 02:25:38.225852 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:38.225481    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:25:38.225852 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:38.225513    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j" event={"ID":"0fabda73-1c04-4032-8a25-2ab7351f572e","Type":"ContainerStarted","Data":"e4091772abdb5b5db12599641f1545a89d809373bf55229cdb15be8ef500ab07"}
Sep 16 02:25:38.245696 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:38.245649    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j" podStartSLOduration=1.245633336 podStartE2EDuration="1.245633336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:25:38.242871514 +0000 UTC m=+437.672932902" watchObservedRunningTime="2026-09-16 02:25:38.245633336 +0000 UTC m=+437.675694721"
Sep 16 02:25:39.114943 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:39.114903    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938ae59d-320d-4b66-aa2d-aff848c5c972" path="/var/lib/kubelet/pods/938ae59d-320d-4b66-aa2d-aff848c5c972/volumes"
Sep 16 02:25:49.234224 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:25:49.234148    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:26:06.331887 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.331850    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gdmkt"]
Sep 16 02:26:06.332362 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.332229    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938ae59d-320d-4b66-aa2d-aff848c5c972" containerName="manager"
Sep 16 02:26:06.332362 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.332242    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="938ae59d-320d-4b66-aa2d-aff848c5c972" containerName="manager"
Sep 16 02:26:06.332362 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.332301    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="938ae59d-320d-4b66-aa2d-aff848c5c972" containerName="manager"
Sep 16 02:26:06.355819 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.355781    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gdmkt"]
Sep 16 02:26:06.355977 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.355908    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.358117 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.358091    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 02:26:06.436787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.436754    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gdmkt"]
Sep 16 02:26:06.456971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.456937    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4n7\" (UniqueName: \"kubernetes.io/projected/6b2313ba-d69d-454d-818d-21380e0ee446-kube-api-access-rb4n7\") pod \"limitador-limitador-8588749977-gdmkt\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") " pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.457239 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.456989    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b2313ba-d69d-454d-818d-21380e0ee446-config-file\") pod \"limitador-limitador-8588749977-gdmkt\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") " pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.558255 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.558217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4n7\" (UniqueName: \"kubernetes.io/projected/6b2313ba-d69d-454d-818d-21380e0ee446-kube-api-access-rb4n7\") pod \"limitador-limitador-8588749977-gdmkt\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") " pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.558424 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.558270    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b2313ba-d69d-454d-818d-21380e0ee446-config-file\") pod \"limitador-limitador-8588749977-gdmkt\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") " pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.558905 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.558888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b2313ba-d69d-454d-818d-21380e0ee446-config-file\") pod \"limitador-limitador-8588749977-gdmkt\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") " pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.565934 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.565909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4n7\" (UniqueName: \"kubernetes.io/projected/6b2313ba-d69d-454d-818d-21380e0ee446-kube-api-access-rb4n7\") pod \"limitador-limitador-8588749977-gdmkt\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") " pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.667418 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.667335    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:06.938668 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:06.938618    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gdmkt"]
Sep 16 02:26:06.940560 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:26:06.940526    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2313ba_d69d_454d_818d_21380e0ee446.slice/crio-e5e45cc909041630479863026d43400fe73f544f85289fe2135b34a9e421a9cf WatchSource:0}: Error finding container e5e45cc909041630479863026d43400fe73f544f85289fe2135b34a9e421a9cf: Status 404 returned error can't find the container with id e5e45cc909041630479863026d43400fe73f544f85289fe2135b34a9e421a9cf
Sep 16 02:26:07.235381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.235309    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-tjrst"]
Sep 16 02:26:07.240143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.240121    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tjrst"
Sep 16 02:26:07.242520 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.242214    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xkv7p\""
Sep 16 02:26:07.244454 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.244408    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tjrst"]
Sep 16 02:26:07.264739 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.264713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6r95\" (UniqueName: \"kubernetes.io/projected/b9e7653c-744f-4eba-9367-34932c325282-kube-api-access-p6r95\") pod \"authorino-59c974bd48-tjrst\" (UID: \"b9e7653c-744f-4eba-9367-34932c325282\") " pod="kuadrant-system/authorino-59c974bd48-tjrst"
Sep 16 02:26:07.365983 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.365946    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6r95\" (UniqueName: \"kubernetes.io/projected/b9e7653c-744f-4eba-9367-34932c325282-kube-api-access-p6r95\") pod \"authorino-59c974bd48-tjrst\" (UID: \"b9e7653c-744f-4eba-9367-34932c325282\") " pod="kuadrant-system/authorino-59c974bd48-tjrst"
Sep 16 02:26:07.372039 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.372008    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt" event={"ID":"6b2313ba-d69d-454d-818d-21380e0ee446","Type":"ContainerStarted","Data":"e5e45cc909041630479863026d43400fe73f544f85289fe2135b34a9e421a9cf"}
Sep 16 02:26:07.373235 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.373212    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6r95\" (UniqueName: \"kubernetes.io/projected/b9e7653c-744f-4eba-9367-34932c325282-kube-api-access-p6r95\") pod \"authorino-59c974bd48-tjrst\" (UID: \"b9e7653c-744f-4eba-9367-34932c325282\") " pod="kuadrant-system/authorino-59c974bd48-tjrst"
Sep 16 02:26:07.551349 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.551251    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tjrst"
Sep 16 02:26:07.818764 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:07.818535    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tjrst"]
Sep 16 02:26:07.821192 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:26:07.821163    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9e7653c_744f_4eba_9367_34932c325282.slice/crio-8db1a474c3c59ce1bd4689c22ad7acef09e6dbab2cc63046ed5d0339cd99e320 WatchSource:0}: Error finding container 8db1a474c3c59ce1bd4689c22ad7acef09e6dbab2cc63046ed5d0339cd99e320: Status 404 returned error can't find the container with id 8db1a474c3c59ce1bd4689c22ad7acef09e6dbab2cc63046ed5d0339cd99e320
Sep 16 02:26:08.378586 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:08.378551    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt" event={"ID":"6b2313ba-d69d-454d-818d-21380e0ee446","Type":"ContainerStarted","Data":"ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061"}
Sep 16 02:26:08.378979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:08.378620    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:08.379743 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:08.379720    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tjrst" event={"ID":"b9e7653c-744f-4eba-9367-34932c325282","Type":"ContainerStarted","Data":"8db1a474c3c59ce1bd4689c22ad7acef09e6dbab2cc63046ed5d0339cd99e320"}
Sep 16 02:26:08.393703 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:08.393662    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt" podStartSLOduration=1.112885628 podStartE2EDuration="2.393647702s" totalImagesPullingTime="1.280762074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:08.391830758 +0000 UTC m=+467.821892145" watchObservedRunningTime="2026-09-16 02:26:08.393647702 +0000 UTC m=+467.823709150"
Sep 16 02:26:10.393688 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:10.393649    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tjrst" event={"ID":"b9e7653c-744f-4eba-9367-34932c325282","Type":"ContainerStarted","Data":"f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272"}
Sep 16 02:26:10.406543 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:10.406467    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-tjrst" podStartSLOduration=1.837451552 podStartE2EDuration="3.406420418s" totalImagesPullingTime="1.568968866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:10.405723328 +0000 UTC m=+469.835784705" watchObservedRunningTime="2026-09-16 02:26:10.406420418 +0000 UTC m=+469.836481805"
Sep 16 02:26:19.386159 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:19.386126    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:20.124349 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.124310    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gdmkt"]
Sep 16 02:26:20.124580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.124553    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt" podUID="6b2313ba-d69d-454d-818d-21380e0ee446" containerName="limitador" containerID="cri-o://ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061" gracePeriod=30
Sep 16 02:26:20.669157 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.669131    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:20.785319 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.785217    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b2313ba-d69d-454d-818d-21380e0ee446-config-file\") pod \"6b2313ba-d69d-454d-818d-21380e0ee446\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") "
Sep 16 02:26:20.785319 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.785310    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb4n7\" (UniqueName: \"kubernetes.io/projected/6b2313ba-d69d-454d-818d-21380e0ee446-kube-api-access-rb4n7\") pod \"6b2313ba-d69d-454d-818d-21380e0ee446\" (UID: \"6b2313ba-d69d-454d-818d-21380e0ee446\") "
Sep 16 02:26:20.785670 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.785650    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2313ba-d69d-454d-818d-21380e0ee446-config-file" (OuterVolumeSpecName: "config-file") pod "6b2313ba-d69d-454d-818d-21380e0ee446" (UID: "6b2313ba-d69d-454d-818d-21380e0ee446"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:26:20.787302 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.787282    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2313ba-d69d-454d-818d-21380e0ee446-kube-api-access-rb4n7" (OuterVolumeSpecName: "kube-api-access-rb4n7") pod "6b2313ba-d69d-454d-818d-21380e0ee446" (UID: "6b2313ba-d69d-454d-818d-21380e0ee446"). InnerVolumeSpecName "kube-api-access-rb4n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:26:20.886384 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.886341    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b2313ba-d69d-454d-818d-21380e0ee446-config-file\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:26:20.886384 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:20.886376    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4n7\" (UniqueName: \"kubernetes.io/projected/6b2313ba-d69d-454d-818d-21380e0ee446-kube-api-access-rb4n7\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:26:21.447902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.447868    2556 generic.go:356] "Generic (PLEG): container finished" podID="6b2313ba-d69d-454d-818d-21380e0ee446" containerID="ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061" exitCode=0
Sep 16 02:26:21.448063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.447910    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt" event={"ID":"6b2313ba-d69d-454d-818d-21380e0ee446","Type":"ContainerDied","Data":"ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061"}
Sep 16 02:26:21.448063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.447914    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt"
Sep 16 02:26:21.448063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.447949    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gdmkt" event={"ID":"6b2313ba-d69d-454d-818d-21380e0ee446","Type":"ContainerDied","Data":"e5e45cc909041630479863026d43400fe73f544f85289fe2135b34a9e421a9cf"}
Sep 16 02:26:21.448063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.447970    2556 scope.go:117] "RemoveContainer" containerID="ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061"
Sep 16 02:26:21.456254 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.456221    2556 scope.go:117] "RemoveContainer" containerID="ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061"
Sep 16 02:26:21.456549 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:26:21.456529    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061\": container with ID starting with ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061 not found: ID does not exist" containerID="ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061"
Sep 16 02:26:21.456607 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.456558    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061"} err="failed to get container status \"ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061\": rpc error: code = NotFound desc = could not find container \"ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061\": container with ID starting with ac54797fcca1a0d710da2553112e3d20c61a36a0e2f7adc567ab319ab3843061 not found: ID does not exist"
Sep 16 02:26:21.462603 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.462582    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gdmkt"]
Sep 16 02:26:21.466346 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:21.466322    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gdmkt"]
Sep 16 02:26:23.115017 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:23.114980    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b2313ba-d69d-454d-818d-21380e0ee446" path="/var/lib/kubelet/pods/6b2313ba-d69d-454d-818d-21380e0ee446/volumes"
Sep 16 02:26:25.001467 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.001413    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s8264"]
Sep 16 02:26:25.001997 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.001976    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2313ba-d69d-454d-818d-21380e0ee446" containerName="limitador"
Sep 16 02:26:25.002041 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.002001    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2313ba-d69d-454d-818d-21380e0ee446" containerName="limitador"
Sep 16 02:26:25.002127 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.002114    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2313ba-d69d-454d-818d-21380e0ee446" containerName="limitador"
Sep 16 02:26:25.006783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.006760    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.009143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.009119    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 02:26:25.009273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.009151    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 02:26:25.009273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.009155    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 02:26:25.009795 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.009780    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wwx8g\""
Sep 16 02:26:25.014770 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.014655    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s8264"]
Sep 16 02:26:25.123448 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.123399    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtl4\" (UniqueName: \"kubernetes.io/projected/bf91f84f-3424-45b6-b480-7f4c0eba9d6b-kube-api-access-vbtl4\") pod \"postgres-69fb4594cf-s8264\" (UID: \"bf91f84f-3424-45b6-b480-7f4c0eba9d6b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.123595 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.123470    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf91f84f-3424-45b6-b480-7f4c0eba9d6b-data\") pod \"postgres-69fb4594cf-s8264\" (UID: \"bf91f84f-3424-45b6-b480-7f4c0eba9d6b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.224476 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.224421    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtl4\" (UniqueName: \"kubernetes.io/projected/bf91f84f-3424-45b6-b480-7f4c0eba9d6b-kube-api-access-vbtl4\") pod \"postgres-69fb4594cf-s8264\" (UID: \"bf91f84f-3424-45b6-b480-7f4c0eba9d6b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.224476 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.224473    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf91f84f-3424-45b6-b480-7f4c0eba9d6b-data\") pod \"postgres-69fb4594cf-s8264\" (UID: \"bf91f84f-3424-45b6-b480-7f4c0eba9d6b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.224900 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.224881    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf91f84f-3424-45b6-b480-7f4c0eba9d6b-data\") pod \"postgres-69fb4594cf-s8264\" (UID: \"bf91f84f-3424-45b6-b480-7f4c0eba9d6b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.232106 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.232074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtl4\" (UniqueName: \"kubernetes.io/projected/bf91f84f-3424-45b6-b480-7f4c0eba9d6b-kube-api-access-vbtl4\") pod \"postgres-69fb4594cf-s8264\" (UID: \"bf91f84f-3424-45b6-b480-7f4c0eba9d6b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.318838 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.318740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:25.594976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:25.594952    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s8264"]
Sep 16 02:26:25.597565 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:26:25.597536    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf91f84f_3424_45b6_b480_7f4c0eba9d6b.slice/crio-ec675a7aeb62aaf956f379cdcbe79072f9c15f714b5498fe282c708f6343db55 WatchSource:0}: Error finding container ec675a7aeb62aaf956f379cdcbe79072f9c15f714b5498fe282c708f6343db55: Status 404 returned error can't find the container with id ec675a7aeb62aaf956f379cdcbe79072f9c15f714b5498fe282c708f6343db55
Sep 16 02:26:26.475831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:26.475754    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264" event={"ID":"bf91f84f-3424-45b6-b480-7f4c0eba9d6b","Type":"ContainerStarted","Data":"ec675a7aeb62aaf956f379cdcbe79072f9c15f714b5498fe282c708f6343db55"}
Sep 16 02:26:31.503695 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:31.503658    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264" event={"ID":"bf91f84f-3424-45b6-b480-7f4c0eba9d6b","Type":"ContainerStarted","Data":"467a645ff008a0be72221241249b44b3e5741ba056325b374a73cf37764d3aa7"}
Sep 16 02:26:31.504076 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:31.503794    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:31.518666 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:31.518620    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264" podStartSLOduration=2.608711597 podStartE2EDuration="7.518607168s" totalImagesPullingTime="4.909895571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:31.516828803 +0000 UTC m=+490.946890194" watchObservedRunningTime="2026-09-16 02:26:31.518607168 +0000 UTC m=+490.948668554"
Sep 16 02:26:37.537190 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:37.537159    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8264"
Sep 16 02:26:38.446551 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.446513    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c699d44ff-ph8bm"]
Sep 16 02:26:38.451683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.451661    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:38.453914 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.453880    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 02:26:38.455604 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.455580    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c699d44ff-ph8bm"]
Sep 16 02:26:38.544082 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.544050    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqfr8\" (UniqueName: \"kubernetes.io/projected/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-kube-api-access-vqfr8\") pod \"authorino-7c699d44ff-ph8bm\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") " pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:38.544082 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.544086    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-tls-cert\") pod \"authorino-7c699d44ff-ph8bm\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") " pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:38.645134 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.645086    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqfr8\" (UniqueName: \"kubernetes.io/projected/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-kube-api-access-vqfr8\") pod \"authorino-7c699d44ff-ph8bm\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") " pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:38.645134 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.645141    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-tls-cert\") pod \"authorino-7c699d44ff-ph8bm\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") " pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:38.647655 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.647631    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-tls-cert\") pod \"authorino-7c699d44ff-ph8bm\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") " pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:38.652036 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.652005    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqfr8\" (UniqueName: \"kubernetes.io/projected/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-kube-api-access-vqfr8\") pod \"authorino-7c699d44ff-ph8bm\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") " pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:38.762033 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:38.761938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:26:39.030526 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.030497    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c699d44ff-ph8bm"]
Sep 16 02:26:39.032557 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:26:39.032529    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad1b15dd_f361_441f_8ff4_ecc028f2f0f2.slice/crio-8c76333fac93f324fb74417f36023b79d968ca52dd55f792281ac55f7065a6ce WatchSource:0}: Error finding container 8c76333fac93f324fb74417f36023b79d968ca52dd55f792281ac55f7065a6ce: Status 404 returned error can't find the container with id 8c76333fac93f324fb74417f36023b79d968ca52dd55f792281ac55f7065a6ce
Sep 16 02:26:39.544390 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.544348    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c699d44ff-ph8bm" event={"ID":"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2","Type":"ContainerStarted","Data":"2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd"}
Sep 16 02:26:39.544799 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.544409    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c699d44ff-ph8bm" event={"ID":"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2","Type":"ContainerStarted","Data":"8c76333fac93f324fb74417f36023b79d968ca52dd55f792281ac55f7065a6ce"}
Sep 16 02:26:39.559651 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.559605    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c699d44ff-ph8bm" podStartSLOduration=1.164276091 podStartE2EDuration="1.559588236s" totalImagesPullingTime="395.312145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:39.557830418 +0000 UTC m=+498.987891805" watchObservedRunningTime="2026-09-16 02:26:39.559588236 +0000 UTC m=+498.989649624"
Sep 16 02:26:39.581275 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.581230    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tjrst"]
Sep 16 02:26:39.581468 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.581411    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-tjrst" podUID="b9e7653c-744f-4eba-9367-34932c325282" containerName="authorino" containerID="cri-o://f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272" gracePeriod=30
Sep 16 02:26:39.824138 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.824114    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tjrst"
Sep 16 02:26:39.955586 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.955487    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6r95\" (UniqueName: \"kubernetes.io/projected/b9e7653c-744f-4eba-9367-34932c325282-kube-api-access-p6r95\") pod \"b9e7653c-744f-4eba-9367-34932c325282\" (UID: \"b9e7653c-744f-4eba-9367-34932c325282\") "
Sep 16 02:26:39.957567 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:39.957537    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e7653c-744f-4eba-9367-34932c325282-kube-api-access-p6r95" (OuterVolumeSpecName: "kube-api-access-p6r95") pod "b9e7653c-744f-4eba-9367-34932c325282" (UID: "b9e7653c-744f-4eba-9367-34932c325282"). InnerVolumeSpecName "kube-api-access-p6r95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:26:40.057087 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.057055    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6r95\" (UniqueName: \"kubernetes.io/projected/b9e7653c-744f-4eba-9367-34932c325282-kube-api-access-p6r95\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:26:40.550850 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.550814    2556 generic.go:356] "Generic (PLEG): container finished" podID="b9e7653c-744f-4eba-9367-34932c325282" containerID="f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272" exitCode=0
Sep 16 02:26:40.551303 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.550857    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tjrst"
Sep 16 02:26:40.551303 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.550889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tjrst" event={"ID":"b9e7653c-744f-4eba-9367-34932c325282","Type":"ContainerDied","Data":"f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272"}
Sep 16 02:26:40.551303 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.550925    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tjrst" event={"ID":"b9e7653c-744f-4eba-9367-34932c325282","Type":"ContainerDied","Data":"8db1a474c3c59ce1bd4689c22ad7acef09e6dbab2cc63046ed5d0339cd99e320"}
Sep 16 02:26:40.551303 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.550941    2556 scope.go:117] "RemoveContainer" containerID="f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272"
Sep 16 02:26:40.559273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.559258    2556 scope.go:117] "RemoveContainer" containerID="f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272"
Sep 16 02:26:40.559545 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:26:40.559527    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272\": container with ID starting with f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272 not found: ID does not exist" containerID="f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272"
Sep 16 02:26:40.559610 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.559556    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272"} err="failed to get container status \"f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272\": rpc error: code = NotFound desc = could not find container \"f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272\": container with ID starting with f646eb4cb1da54416b05accf719e82269882f25bdfcdd8e310889c6f3e673272 not found: ID does not exist"
Sep 16 02:26:40.570024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.570001    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tjrst"]
Sep 16 02:26:40.574411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:40.574387    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tjrst"]
Sep 16 02:26:41.115067 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:41.115038    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e7653c-744f-4eba-9367-34932c325282" path="/var/lib/kubelet/pods/b9e7653c-744f-4eba-9367-34932c325282/volumes"
Sep 16 02:26:44.581863 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.581829    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f594795b-snp5s"]
Sep 16 02:26:44.582248 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.582188    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e7653c-744f-4eba-9367-34932c325282" containerName="authorino"
Sep 16 02:26:44.582248 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.582200    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e7653c-744f-4eba-9367-34932c325282" containerName="authorino"
Sep 16 02:26:44.582315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.582285    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e7653c-744f-4eba-9367-34932c325282" containerName="authorino"
Sep 16 02:26:44.594297 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.594257    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.594894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.594870    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f594795b-snp5s"]
Sep 16 02:26:44.597502 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.597473    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 02:26:44.597637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.597616    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 02:26:44.597739 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.597711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 02:26:44.597880 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.597837    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rvmxp\""
Sep 16 02:26:44.695385 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.695338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-metrics-certs\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.695571 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.695489    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-webhook-cert\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.695571 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.695562    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8g69\" (UniqueName: \"kubernetes.io/projected/40ac788e-06cb-4bab-840e-795d0f9df283-kube-api-access-b8g69\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.796875 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.796836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8g69\" (UniqueName: \"kubernetes.io/projected/40ac788e-06cb-4bab-840e-795d0f9df283-kube-api-access-b8g69\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.797053 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.796894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-metrics-certs\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.797053 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.796952    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-webhook-cert\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.799356 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.799330    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-metrics-certs\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.799490 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.799368    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-webhook-cert\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.805789 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.805764    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8g69\" (UniqueName: \"kubernetes.io/projected/40ac788e-06cb-4bab-840e-795d0f9df283-kube-api-access-b8g69\") pod \"maas-controller-6f594795b-snp5s\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") " pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:44.906581 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:44.906494    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:45.211712 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:45.211685    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f594795b-snp5s"]
Sep 16 02:26:45.213633 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:26:45.213605    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ac788e_06cb_4bab_840e_795d0f9df283.slice/crio-a6f2dbf68b330afb667d4a96769cc3900eea58ed386ead986eee18adee7436ea WatchSource:0}: Error finding container a6f2dbf68b330afb667d4a96769cc3900eea58ed386ead986eee18adee7436ea: Status 404 returned error can't find the container with id a6f2dbf68b330afb667d4a96769cc3900eea58ed386ead986eee18adee7436ea
Sep 16 02:26:45.581781 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:45.581697    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f594795b-snp5s" event={"ID":"40ac788e-06cb-4bab-840e-795d0f9df283","Type":"ContainerStarted","Data":"a6f2dbf68b330afb667d4a96769cc3900eea58ed386ead986eee18adee7436ea"}
Sep 16 02:26:47.594956 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:47.594914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f594795b-snp5s" event={"ID":"40ac788e-06cb-4bab-840e-795d0f9df283","Type":"ContainerStarted","Data":"cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee"}
Sep 16 02:26:47.595315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:47.594980    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:26:47.610495 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:47.610420    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f594795b-snp5s" podStartSLOduration=1.2980961180000001 podStartE2EDuration="3.610403363s" totalImagesPullingTime="2.312307245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:26:47.608881234 +0000 UTC m=+507.038942619" watchObservedRunningTime="2026-09-16 02:26:47.610403363 +0000 UTC m=+507.040464748"
Sep 16 02:26:58.602130 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:26:58.602096    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:27:19.026582 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.026509    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"]
Sep 16 02:27:19.031853 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.031835    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.034526 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.034492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 02:27:19.034660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.034603    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cqzjq\""
Sep 16 02:27:19.034725 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.034607    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 02:27:19.040321 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.040066    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"]
Sep 16 02:27:19.189024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.188988    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-metrics-certs\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.189206 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.189045    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-maas-api-tls\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.189206 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.189075    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1582b0-3fb2-441d-9947-953abaa56804-openshift-service-ca\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.189206 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.189127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fjs\" (UniqueName: \"kubernetes.io/projected/9d1582b0-3fb2-441d-9947-953abaa56804-kube-api-access-p2fjs\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.290135 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.290053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-metrics-certs\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.290135 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.290097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-maas-api-tls\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.290135 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.290115    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1582b0-3fb2-441d-9947-953abaa56804-openshift-service-ca\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.290422 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.290158    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fjs\" (UniqueName: \"kubernetes.io/projected/9d1582b0-3fb2-441d-9947-953abaa56804-kube-api-access-p2fjs\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.290422 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:27:19.290210    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 16 02:27:19.290422 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:27:19.290292    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-metrics-certs podName:9d1582b0-3fb2-441d-9947-953abaa56804 nodeName:}" failed. No retries permitted until 2026-09-16 02:27:19.790275355 +0000 UTC m=+539.220336725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-metrics-certs") pod "maas-api-59f6566bbc-z2w49" (UID: "9d1582b0-3fb2-441d-9947-953abaa56804") : secret "maas-api-metrics-tls" not found
Sep 16 02:27:19.290953 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.290928    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1582b0-3fb2-441d-9947-953abaa56804-openshift-service-ca\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.292755 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.292732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-maas-api-tls\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.297800 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.297779    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fjs\" (UniqueName: \"kubernetes.io/projected/9d1582b0-3fb2-441d-9947-953abaa56804-kube-api-access-p2fjs\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.794759 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.794710    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-metrics-certs\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.797600 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.797574    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1582b0-3fb2-441d-9947-953abaa56804-metrics-certs\") pod \"maas-api-59f6566bbc-z2w49\" (UID: \"9d1582b0-3fb2-441d-9947-953abaa56804\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:19.944920 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:19.944883    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:20.217758 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:20.217726    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"]
Sep 16 02:27:20.219822 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:27:20.219787    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1582b0_3fb2_441d_9947_953abaa56804.slice/crio-e520f2a0e57dd5b6025a7ff349572a6de6ce1e4aeb811e28a75fa15f7a803dac WatchSource:0}: Error finding container e520f2a0e57dd5b6025a7ff349572a6de6ce1e4aeb811e28a75fa15f7a803dac: Status 404 returned error can't find the container with id e520f2a0e57dd5b6025a7ff349572a6de6ce1e4aeb811e28a75fa15f7a803dac
Sep 16 02:27:20.755603 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:20.755564    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49" event={"ID":"9d1582b0-3fb2-441d-9947-953abaa56804","Type":"ContainerStarted","Data":"e520f2a0e57dd5b6025a7ff349572a6de6ce1e4aeb811e28a75fa15f7a803dac"}
Sep 16 02:27:22.766794 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:22.766758    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49" event={"ID":"9d1582b0-3fb2-441d-9947-953abaa56804","Type":"ContainerStarted","Data":"59b58b2f8c87fd0863b7069ead4464a6f1b8819f6b5498083d0a554ab18824ff"}
Sep 16 02:27:22.767170 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:22.766880    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:22.783781 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:22.783726    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49" podStartSLOduration=2.247098952 podStartE2EDuration="3.783709307s" totalImagesPullingTime="1.536610355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:27:22.781388174 +0000 UTC m=+542.211449560" watchObservedRunningTime="2026-09-16 02:27:22.783709307 +0000 UTC m=+542.213770692"
Sep 16 02:27:28.777405 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:28.777377    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-z2w49"
Sep 16 02:27:50.745722 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.745679    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"]
Sep 16 02:27:50.758734 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.758703    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"]
Sep 16 02:27:50.758870 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.758804    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.761257 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.761235    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 02:27:50.761471 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.761239    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 02:27:50.761597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.761296    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f9flb\""
Sep 16 02:27:50.762231 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.762203    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 02:27:50.861796 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.861759    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.861796 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.861796    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lfs\" (UniqueName: \"kubernetes.io/projected/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-kube-api-access-m5lfs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.861984 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.861834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.861984 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.861855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.861984 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.861870    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.861984 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.861897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.962682 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.962650    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.962843 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.962693    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.962843 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.962711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.962843 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.962747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.962843 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.962804    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.962843 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.962825    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lfs\" (UniqueName: \"kubernetes.io/projected/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-kube-api-access-m5lfs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.963114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.963052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.963175 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.963137    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.963219 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.963194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.964919 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.964895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.965204 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.965184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:50.970184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:50.970160    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lfs\" (UniqueName: \"kubernetes.io/projected/b1cee329-df4b-4230-bdcf-0a8a6f27ec03-kube-api-access-m5lfs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68\" (UID: \"b1cee329-df4b-4230-bdcf-0a8a6f27ec03\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:51.070392 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:51.070327    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:27:51.343489 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:51.343458    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"]
Sep 16 02:27:51.345511 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:27:51.345485    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cee329_df4b_4230_bdcf_0a8a6f27ec03.slice/crio-574ce1c42d9270fb7c9d44e38638fdc40479a91ef562b3adf1adcfdfcd77f6a0 WatchSource:0}: Error finding container 574ce1c42d9270fb7c9d44e38638fdc40479a91ef562b3adf1adcfdfcd77f6a0: Status 404 returned error can't find the container with id 574ce1c42d9270fb7c9d44e38638fdc40479a91ef562b3adf1adcfdfcd77f6a0
Sep 16 02:27:51.909311 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:51.909264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68" event={"ID":"b1cee329-df4b-4230-bdcf-0a8a6f27ec03","Type":"ContainerStarted","Data":"574ce1c42d9270fb7c9d44e38638fdc40479a91ef562b3adf1adcfdfcd77f6a0"}
Sep 16 02:27:52.915722 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:52.915682    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68" event={"ID":"b1cee329-df4b-4230-bdcf-0a8a6f27ec03","Type":"ContainerStarted","Data":"fbbc4003c22623631a81890911ca512971ca97a013f0fd034d39b87f6a424306"}
Sep 16 02:27:52.932780 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:27:52.932737    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68" podStartSLOduration=1.943370517 podStartE2EDuration="2.932722957s" totalImagesPullingTime="989.35244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:27:52.930184488 +0000 UTC m=+572.360245875" watchObservedRunningTime="2026-09-16 02:27:52.932722957 +0000 UTC m=+572.362784343"
Sep 16 02:28:01.070787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:01.070753    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:28:01.071283 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:01.070844    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:28:01.083225 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:01.083204    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:28:01.974918 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:01.974887    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dgt68"
Sep 16 02:28:02.947890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:02.947857    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"]
Sep 16 02:28:02.951689 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:02.951667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:02.953732 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:02.953711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 02:28:02.959578 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:02.959555    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"]
Sep 16 02:28:03.065705 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.065669    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76ec54e-5442-402e-95f6-b6a68437b735-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.065865 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.065789    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.065865 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.065842    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.065865 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.065862    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.066002 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.065891    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.066059 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.066043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldjgd\" (UniqueName: \"kubernetes.io/projected/d76ec54e-5442-402e-95f6-b6a68437b735-kube-api-access-ldjgd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167177 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167140    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167360 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167186    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167360 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167294    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167360 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167579 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167368    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjgd\" (UniqueName: \"kubernetes.io/projected/d76ec54e-5442-402e-95f6-b6a68437b735-kube-api-access-ldjgd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167579 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167420    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76ec54e-5442-402e-95f6-b6a68437b735-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167607    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167693 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167682    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.167788 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.167768    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.169626 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.169597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76ec54e-5442-402e-95f6-b6a68437b735-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.170003 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.169982    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76ec54e-5442-402e-95f6-b6a68437b735-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.174847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.174823    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldjgd\" (UniqueName: \"kubernetes.io/projected/d76ec54e-5442-402e-95f6-b6a68437b735-kube-api-access-ldjgd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5bzdz\" (UID: \"d76ec54e-5442-402e-95f6-b6a68437b735\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.263166 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.263100    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:03.532938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.532874    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"]
Sep 16 02:28:03.535106 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:28:03.535081    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76ec54e_5442_402e_95f6_b6a68437b735.slice/crio-7319b3341aa30e7e17c32c0de9fa927201e3e96c14a0707196bce2b8c2ff52a6 WatchSource:0}: Error finding container 7319b3341aa30e7e17c32c0de9fa927201e3e96c14a0707196bce2b8c2ff52a6: Status 404 returned error can't find the container with id 7319b3341aa30e7e17c32c0de9fa927201e3e96c14a0707196bce2b8c2ff52a6
Sep 16 02:28:03.975628 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.975588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz" event={"ID":"d76ec54e-5442-402e-95f6-b6a68437b735","Type":"ContainerStarted","Data":"ed839cc7463c6458347176aa9b8e26c95ed987b187a17bd96f111d0d68f350fa"}
Sep 16 02:28:03.976009 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.975650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz" event={"ID":"d76ec54e-5442-402e-95f6-b6a68437b735","Type":"ContainerStarted","Data":"7319b3341aa30e7e17c32c0de9fa927201e3e96c14a0707196bce2b8c2ff52a6"}
Sep 16 02:28:03.991067 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:03.991022    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz" podStartSLOduration=1.7919861030000002 podStartE2EDuration="1.991007166s" totalImagesPullingTime="199.021063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:28:03.990147318 +0000 UTC m=+583.420208707" watchObservedRunningTime="2026-09-16 02:28:03.991007166 +0000 UTC m=+583.421068551"
Sep 16 02:28:13.264303 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:13.264270    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:13.264814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:13.264325    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:13.276508 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:13.276487    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:14.037384 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:14.037352    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5bzdz"
Sep 16 02:28:20.961287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:20.961257    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:28:20.962072 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:28:20.962049    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:29:40.071683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.071649    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c699d44ff-ph8bm"]
Sep 16 02:29:40.072249 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.071830    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7c699d44ff-ph8bm" podUID="ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" containerName="authorino" containerID="cri-o://2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd" gracePeriod=30
Sep 16 02:29:40.314349 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.314323    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:29:40.386581 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.386497    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-tls-cert\") pod \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") "
Sep 16 02:29:40.386735 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.386598    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqfr8\" (UniqueName: \"kubernetes.io/projected/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-kube-api-access-vqfr8\") pod \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\" (UID: \"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2\") "
Sep 16 02:29:40.388533 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.388505    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-kube-api-access-vqfr8" (OuterVolumeSpecName: "kube-api-access-vqfr8") pod "ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" (UID: "ad1b15dd-f361-441f-8ff4-ecc028f2f0f2"). InnerVolumeSpecName "kube-api-access-vqfr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:29:40.397281 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.397256    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" (UID: "ad1b15dd-f361-441f-8ff4-ecc028f2f0f2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:29:40.435600 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.435573    2556 generic.go:356] "Generic (PLEG): container finished" podID="ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" containerID="2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd" exitCode=0
Sep 16 02:29:40.435711 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.435624    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c699d44ff-ph8bm"
Sep 16 02:29:40.435711 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.435661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c699d44ff-ph8bm" event={"ID":"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2","Type":"ContainerDied","Data":"2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd"}
Sep 16 02:29:40.435822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.435709    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c699d44ff-ph8bm" event={"ID":"ad1b15dd-f361-441f-8ff4-ecc028f2f0f2","Type":"ContainerDied","Data":"8c76333fac93f324fb74417f36023b79d968ca52dd55f792281ac55f7065a6ce"}
Sep 16 02:29:40.435822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.435734    2556 scope.go:117] "RemoveContainer" containerID="2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd"
Sep 16 02:29:40.445124 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.445104    2556 scope.go:117] "RemoveContainer" containerID="2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd"
Sep 16 02:29:40.445381 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:29:40.445363    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd\": container with ID starting with 2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd not found: ID does not exist" containerID="2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd"
Sep 16 02:29:40.445461 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.445393    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd"} err="failed to get container status \"2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd\": rpc error: code = NotFound desc = could not find container \"2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd\": container with ID starting with 2c108c6a0532875658521f7481bdfc939e9c0cd411a45705bf480eacf1957ddd not found: ID does not exist"
Sep 16 02:29:40.455787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.455767    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c699d44ff-ph8bm"]
Sep 16 02:29:40.457631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.457610    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c699d44ff-ph8bm"]
Sep 16 02:29:40.487750 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.487729    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-tls-cert\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:29:40.487750 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.487748    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqfr8\" (UniqueName: \"kubernetes.io/projected/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2-kube-api-access-vqfr8\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:29:40.678751 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.678717    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f594795b-snp5s"]
Sep 16 02:29:40.679008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.678966    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f594795b-snp5s" podUID="40ac788e-06cb-4bab-840e-795d0f9df283" containerName="manager" containerID="cri-o://cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee" gracePeriod=30
Sep 16 02:29:40.921169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.921145    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:29:40.992705 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.992620    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8g69\" (UniqueName: \"kubernetes.io/projected/40ac788e-06cb-4bab-840e-795d0f9df283-kube-api-access-b8g69\") pod \"40ac788e-06cb-4bab-840e-795d0f9df283\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") "
Sep 16 02:29:40.992705 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.992657    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-webhook-cert\") pod \"40ac788e-06cb-4bab-840e-795d0f9df283\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") "
Sep 16 02:29:40.992705 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.992683    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-metrics-certs\") pod \"40ac788e-06cb-4bab-840e-795d0f9df283\" (UID: \"40ac788e-06cb-4bab-840e-795d0f9df283\") "
Sep 16 02:29:40.994745 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.994706    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "40ac788e-06cb-4bab-840e-795d0f9df283" (UID: "40ac788e-06cb-4bab-840e-795d0f9df283"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:29:40.994868 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.994852    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ac788e-06cb-4bab-840e-795d0f9df283-kube-api-access-b8g69" (OuterVolumeSpecName: "kube-api-access-b8g69") pod "40ac788e-06cb-4bab-840e-795d0f9df283" (UID: "40ac788e-06cb-4bab-840e-795d0f9df283"). InnerVolumeSpecName "kube-api-access-b8g69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:29:40.995120 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:40.995082    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "40ac788e-06cb-4bab-840e-795d0f9df283" (UID: "40ac788e-06cb-4bab-840e-795d0f9df283"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:29:41.094278 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.094249    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-webhook-cert\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:29:41.094278 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.094276    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40ac788e-06cb-4bab-840e-795d0f9df283-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:29:41.094702 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.094287    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8g69\" (UniqueName: \"kubernetes.io/projected/40ac788e-06cb-4bab-840e-795d0f9df283-kube-api-access-b8g69\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:29:41.114590 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.114563    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" path="/var/lib/kubelet/pods/ad1b15dd-f361-441f-8ff4-ecc028f2f0f2/volumes"
Sep 16 02:29:41.442978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.442947    2556 generic.go:356] "Generic (PLEG): container finished" podID="40ac788e-06cb-4bab-840e-795d0f9df283" containerID="cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee" exitCode=0
Sep 16 02:29:41.443144 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.442990    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f594795b-snp5s"
Sep 16 02:29:41.443144 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.443018    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f594795b-snp5s" event={"ID":"40ac788e-06cb-4bab-840e-795d0f9df283","Type":"ContainerDied","Data":"cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee"}
Sep 16 02:29:41.443144 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.443056    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f594795b-snp5s" event={"ID":"40ac788e-06cb-4bab-840e-795d0f9df283","Type":"ContainerDied","Data":"a6f2dbf68b330afb667d4a96769cc3900eea58ed386ead986eee18adee7436ea"}
Sep 16 02:29:41.443144 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.443078    2556 scope.go:117] "RemoveContainer" containerID="cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee"
Sep 16 02:29:41.451597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.451577    2556 scope.go:117] "RemoveContainer" containerID="cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee"
Sep 16 02:29:41.451844 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:29:41.451826    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee\": container with ID starting with cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee not found: ID does not exist" containerID="cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee"
Sep 16 02:29:41.451889 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.451850    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee"} err="failed to get container status \"cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee\": rpc error: code = NotFound desc = could not find container \"cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee\": container with ID starting with cd22c6f3d3fac9cc436e6d842412680f5a25f2b70d91737daf64fff15c3229ee not found: ID does not exist"
Sep 16 02:29:41.457776 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.457753    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f594795b-snp5s"]
Sep 16 02:29:41.463385 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:41.463366    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f594795b-snp5s"]
Sep 16 02:29:43.115096 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:43.115065    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ac788e-06cb-4bab-840e-795d0f9df283" path="/var/lib/kubelet/pods/40ac788e-06cb-4bab-840e-795d0f9df283/volumes"
Sep 16 02:29:57.389181 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.389150    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"]
Sep 16 02:29:57.389715 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.389684    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" containerName="authorino"
Sep 16 02:29:57.389715 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.389708    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" containerName="authorino"
Sep 16 02:29:57.389715 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.389717    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ac788e-06cb-4bab-840e-795d0f9df283" containerName="manager"
Sep 16 02:29:57.389972 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.389723    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ac788e-06cb-4bab-840e-795d0f9df283" containerName="manager"
Sep 16 02:29:57.389972 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.389808    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad1b15dd-f361-441f-8ff4-ecc028f2f0f2" containerName="authorino"
Sep 16 02:29:57.389972 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.389822    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ac788e-06cb-4bab-840e-795d0f9df283" containerName="manager"
Sep 16 02:29:57.392974 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.392955    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.395345 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.395324    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7v8jt\""
Sep 16 02:29:57.395345 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.395338    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 02:29:57.405713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.405689    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"]
Sep 16 02:29:57.538318 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.538289    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9582f82-fc4a-448c-99e4-4fc2fc2d118d-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-p6mx9\" (UID: \"c9582f82-fc4a-448c-99e4-4fc2fc2d118d\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.538494 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.538370    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjshn\" (UniqueName: \"kubernetes.io/projected/c9582f82-fc4a-448c-99e4-4fc2fc2d118d-kube-api-access-fjshn\") pod \"payload-processing-5d7dbbd56b-p6mx9\" (UID: \"c9582f82-fc4a-448c-99e4-4fc2fc2d118d\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.639176 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.639146    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9582f82-fc4a-448c-99e4-4fc2fc2d118d-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-p6mx9\" (UID: \"c9582f82-fc4a-448c-99e4-4fc2fc2d118d\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.639341 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.639218    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjshn\" (UniqueName: \"kubernetes.io/projected/c9582f82-fc4a-448c-99e4-4fc2fc2d118d-kube-api-access-fjshn\") pod \"payload-processing-5d7dbbd56b-p6mx9\" (UID: \"c9582f82-fc4a-448c-99e4-4fc2fc2d118d\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.639995 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.639975    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9582f82-fc4a-448c-99e4-4fc2fc2d118d-plugins-config-volume\") pod \"payload-processing-5d7dbbd56b-p6mx9\" (UID: \"c9582f82-fc4a-448c-99e4-4fc2fc2d118d\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.646377 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.646354    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjshn\" (UniqueName: \"kubernetes.io/projected/c9582f82-fc4a-448c-99e4-4fc2fc2d118d-kube-api-access-fjshn\") pod \"payload-processing-5d7dbbd56b-p6mx9\" (UID: \"c9582f82-fc4a-448c-99e4-4fc2fc2d118d\") " pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.703105 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.703082    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:29:57.970710 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.970684    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"]
Sep 16 02:29:57.976075 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:29:57.976041    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9582f82_fc4a_448c_99e4_4fc2fc2d118d.slice/crio-3947830e03ad3c381b0101cd94461103c4286462bf3b0c04746528f703c9f254 WatchSource:0}: Error finding container 3947830e03ad3c381b0101cd94461103c4286462bf3b0c04746528f703c9f254: Status 404 returned error can't find the container with id 3947830e03ad3c381b0101cd94461103c4286462bf3b0c04746528f703c9f254
Sep 16 02:29:57.978142 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:57.978124    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:29:58.526658 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:58.526618    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" event={"ID":"c9582f82-fc4a-448c-99e4-4fc2fc2d118d","Type":"ContainerStarted","Data":"3947830e03ad3c381b0101cd94461103c4286462bf3b0c04746528f703c9f254"}
Sep 16 02:29:59.538993 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:59.538954    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" event={"ID":"c9582f82-fc4a-448c-99e4-4fc2fc2d118d","Type":"ContainerStarted","Data":"be38ed01d095b652915db987b4a2d1dcc81a83620ec3dbde3a2b0778deba7147"}
Sep 16 02:29:59.539455 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:29:59.539099    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:30:08.943801 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:08.943746    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" podStartSLOduration=10.854235671 podStartE2EDuration="11.943729203s" totalImagesPullingTime="1.089493532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:29:59.552919231 +0000 UTC m=+698.982980616" watchObservedRunningTime="2026-09-16 02:30:08.943729203 +0000 UTC m=+708.373790588"
Sep 16 02:30:08.944887 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:08.944866    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"]
Sep 16 02:30:08.948389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:08.948367    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"
Sep 16 02:30:08.950614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:08.950589    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 16 02:30:08.950910 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:08.950894    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-6757b\""
Sep 16 02:30:08.957639 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:08.957618    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"]
Sep 16 02:30:09.042050 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:09.042011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfmf\" (UniqueName: \"kubernetes.io/projected/c6f201a9-82a7-4833-bf0b-1e704d0ae94e-kube-api-access-jqfmf\") pod \"ai-gateway-controller-5dbb7f5f95-zwblq\" (UID: \"c6f201a9-82a7-4833-bf0b-1e704d0ae94e\") " pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"
Sep 16 02:30:09.142820 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:09.142792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfmf\" (UniqueName: \"kubernetes.io/projected/c6f201a9-82a7-4833-bf0b-1e704d0ae94e-kube-api-access-jqfmf\") pod \"ai-gateway-controller-5dbb7f5f95-zwblq\" (UID: \"c6f201a9-82a7-4833-bf0b-1e704d0ae94e\") " pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"
Sep 16 02:30:09.150382 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:09.150352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfmf\" (UniqueName: \"kubernetes.io/projected/c6f201a9-82a7-4833-bf0b-1e704d0ae94e-kube-api-access-jqfmf\") pod \"ai-gateway-controller-5dbb7f5f95-zwblq\" (UID: \"c6f201a9-82a7-4833-bf0b-1e704d0ae94e\") " pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"
Sep 16 02:30:09.259667 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:09.259588    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"
Sep 16 02:30:09.525981 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:09.525959    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"]
Sep 16 02:30:09.528026 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:30:09.527995    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f201a9_82a7_4833_bf0b_1e704d0ae94e.slice/crio-1c1a636de3ef5d651032bbdd4c23f075eb38380e1929083d1b955013b0269339 WatchSource:0}: Error finding container 1c1a636de3ef5d651032bbdd4c23f075eb38380e1929083d1b955013b0269339: Status 404 returned error can't find the container with id 1c1a636de3ef5d651032bbdd4c23f075eb38380e1929083d1b955013b0269339
Sep 16 02:30:09.585498 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:09.585466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq" event={"ID":"c6f201a9-82a7-4833-bf0b-1e704d0ae94e","Type":"ContainerStarted","Data":"1c1a636de3ef5d651032bbdd4c23f075eb38380e1929083d1b955013b0269339"}
Sep 16 02:30:10.545907 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:10.545876    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9"
Sep 16 02:30:10.546397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:10.546363    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:30:11.599485 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:11.599447    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq" event={"ID":"c6f201a9-82a7-4833-bf0b-1e704d0ae94e","Type":"ContainerStarted","Data":"83b8ec78d71a7d11d926baf4703d80308a0fd791f67d6adf235fb2046cc91e9a"}
Sep 16 02:30:11.599960 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:11.599569    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"
Sep 16 02:30:11.617455 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:11.614610    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq" podStartSLOduration=2.3720497050000002 podStartE2EDuration="3.614586746s" totalImagesPullingTime="1.242537041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:11.613703837 +0000 UTC m=+711.043765224" watchObservedRunningTime="2026-09-16 02:30:11.614586746 +0000 UTC m=+711.044648133"
Sep 16 02:30:22.607205 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:22.607173    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5dbb7f5f95-zwblq"
Sep 16 02:30:51.242937 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.242902    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"]
Sep 16 02:30:51.246819 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.246797    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.249024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.249001    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6dfb47-serving-cert\""
Sep 16 02:30:51.257575 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.257549    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"]
Sep 16 02:30:51.399646 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.399602    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2632151a-ead4-4d89-a482-f788058d0545-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.399854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.399661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65w8\" (UniqueName: \"kubernetes.io/projected/2632151a-ead4-4d89-a482-f788058d0545-kube-api-access-m65w8\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.399854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.399751    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-maas-api-tls\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.399854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.399809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-metrics-certs\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.500788 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.500691    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2632151a-ead4-4d89-a482-f788058d0545-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.500788 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.500766    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65w8\" (UniqueName: \"kubernetes.io/projected/2632151a-ead4-4d89-a482-f788058d0545-kube-api-access-m65w8\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.501010 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.500795    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-maas-api-tls\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.501010 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.500819    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-metrics-certs\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.501540 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.501512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2632151a-ead4-4d89-a482-f788058d0545-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.503468 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.503420    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-metrics-certs\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.503696 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.503677    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-maas-api-tls\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.510101 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.510069    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65w8\" (UniqueName: \"kubernetes.io/projected/2632151a-ead4-4d89-a482-f788058d0545-kube-api-access-m65w8\") pod \"maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.560815 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.560786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:51.845210 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:51.845185    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"]
Sep 16 02:30:51.846810 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:30:51.846784    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2632151a_ead4_4d89_a482_f788058d0545.slice/crio-4fb70bad6776ba8b0d5af3a13011bc564f8b597ce7ab841fdc9d73b1111e4902 WatchSource:0}: Error finding container 4fb70bad6776ba8b0d5af3a13011bc564f8b597ce7ab841fdc9d73b1111e4902: Status 404 returned error can't find the container with id 4fb70bad6776ba8b0d5af3a13011bc564f8b597ce7ab841fdc9d73b1111e4902
Sep 16 02:30:52.213559 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.213525    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"]
Sep 16 02:30:52.218949 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.218923    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.221185 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.221160    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-a3d7eb-serving-cert\""
Sep 16 02:30:52.226117 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.226093    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"]
Sep 16 02:30:52.308005 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.307965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.308411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.308014    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f8e17e-7c2a-4415-b101-a8357bd0ac33-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.308411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.308128    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mklx\" (UniqueName: \"kubernetes.io/projected/90f8e17e-7c2a-4415-b101-a8357bd0ac33-kube-api-access-2mklx\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.308411 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.308172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-metrics-certs\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.409229 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.409197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.409325 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.409261    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f8e17e-7c2a-4415-b101-a8357bd0ac33-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.409325 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.409315    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mklx\" (UniqueName: \"kubernetes.io/projected/90f8e17e-7c2a-4415-b101-a8357bd0ac33-kube-api-access-2mklx\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.409449 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:30:52.409340    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-serving-cert: secret "maas-api-e2e-worker-w3-a3d7eb-serving-cert" not found
Sep 16 02:30:52.409449 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.409354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-metrics-certs\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.409449 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:30:52.409406    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls podName:90f8e17e-7c2a-4415-b101-a8357bd0ac33 nodeName:}" failed. No retries permitted until 2026-09-16 02:30:52.909383905 +0000 UTC m=+752.339445282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls") pod "maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q" (UID: "90f8e17e-7c2a-4415-b101-a8357bd0ac33") : secret "maas-api-e2e-worker-w3-a3d7eb-serving-cert" not found
Sep 16 02:30:52.410067 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.410043    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f8e17e-7c2a-4415-b101-a8357bd0ac33-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.411957 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.411935    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-metrics-certs\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.417868 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.417847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mklx\" (UniqueName: \"kubernetes.io/projected/90f8e17e-7c2a-4415-b101-a8357bd0ac33-kube-api-access-2mklx\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.804600 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.804560    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b" event={"ID":"2632151a-ead4-4d89-a482-f788058d0545","Type":"ContainerStarted","Data":"e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488"}
Sep 16 02:30:52.804795 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.804636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b" event={"ID":"2632151a-ead4-4d89-a482-f788058d0545","Type":"ContainerStarted","Data":"4fb70bad6776ba8b0d5af3a13011bc564f8b597ce7ab841fdc9d73b1111e4902"}
Sep 16 02:30:52.804795 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.804682    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:30:52.821383 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.821334    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b" podStartSLOduration=1.272151591 podStartE2EDuration="1.82131358s" totalImagesPullingTime="549.161989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:52.818849806 +0000 UTC m=+752.248911193" watchObservedRunningTime="2026-09-16 02:30:52.82131358 +0000 UTC m=+752.251374999"
Sep 16 02:30:52.915059 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.915018    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:52.917731 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:52.917703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:53.128489 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.128382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"]
Sep 16 02:30:53.132370 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.132346    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.132706 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.132681    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:53.134589 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.134565    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-5b59aa-serving-cert\""
Sep 16 02:30:53.144918 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.144611    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"]
Sep 16 02:30:53.218221 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.218155    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-metrics-certs\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.218221 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.218218    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjck\" (UniqueName: \"kubernetes.io/projected/5642889a-1592-4949-9fcd-215e3656111b-kube-api-access-qtjck\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.218403 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.218266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.218579 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.218558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5642889a-1592-4949-9fcd-215e3656111b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.319864 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.319824    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5642889a-1592-4949-9fcd-215e3656111b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.320337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.319876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-metrics-certs\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.320337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.319906    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjck\" (UniqueName: \"kubernetes.io/projected/5642889a-1592-4949-9fcd-215e3656111b-kube-api-access-qtjck\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.320337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.319931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.320337 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:30:53.320074    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-serving-cert: secret "maas-api-e2e-worker-w0-5b59aa-serving-cert" not found
Sep 16 02:30:53.320337 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:30:53.320142    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls podName:5642889a-1592-4949-9fcd-215e3656111b nodeName:}" failed. No retries permitted until 2026-09-16 02:30:53.820121906 +0000 UTC m=+753.250183277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls") pod "maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms" (UID: "5642889a-1592-4949-9fcd-215e3656111b") : secret "maas-api-e2e-worker-w0-5b59aa-serving-cert" not found
Sep 16 02:30:53.320709 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.320687    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5642889a-1592-4949-9fcd-215e3656111b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.322694 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.322670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-metrics-certs\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.337979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.337950    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjck\" (UniqueName: \"kubernetes.io/projected/5642889a-1592-4949-9fcd-215e3656111b-kube-api-access-qtjck\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.418851 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.418823    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"]
Sep 16 02:30:53.420807 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:30:53.420781    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f8e17e_7c2a_4415_b101_a8357bd0ac33.slice/crio-ef904cdf1fe13a8da329bd99aa41ec50c218f3a7cc84e3592beca9e87d569f8a WatchSource:0}: Error finding container ef904cdf1fe13a8da329bd99aa41ec50c218f3a7cc84e3592beca9e87d569f8a: Status 404 returned error can't find the container with id ef904cdf1fe13a8da329bd99aa41ec50c218f3a7cc84e3592beca9e87d569f8a
Sep 16 02:30:53.812994 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.812941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q" event={"ID":"90f8e17e-7c2a-4415-b101-a8357bd0ac33","Type":"ContainerStarted","Data":"ef904cdf1fe13a8da329bd99aa41ec50c218f3a7cc84e3592beca9e87d569f8a"}
Sep 16 02:30:53.823800 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.823769    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:53.826300 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:53.826275    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:54.065783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.065702    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:54.339816 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.339794    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"]
Sep 16 02:30:54.341956 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:30:54.341929    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5642889a_1592_4949_9fcd_215e3656111b.slice/crio-c7f69e888ab984a7f24cf4b05188deb306c83434a3f40d65f2016f3200b14a44 WatchSource:0}: Error finding container c7f69e888ab984a7f24cf4b05188deb306c83434a3f40d65f2016f3200b14a44: Status 404 returned error can't find the container with id c7f69e888ab984a7f24cf4b05188deb306c83434a3f40d65f2016f3200b14a44
Sep 16 02:30:54.819617 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.819577    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms" event={"ID":"5642889a-1592-4949-9fcd-215e3656111b","Type":"ContainerStarted","Data":"05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f"}
Sep 16 02:30:54.819801 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.819758    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:30:54.819872 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.819799    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms" event={"ID":"5642889a-1592-4949-9fcd-215e3656111b","Type":"ContainerStarted","Data":"c7f69e888ab984a7f24cf4b05188deb306c83434a3f40d65f2016f3200b14a44"}
Sep 16 02:30:54.821095 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.821073    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q" event={"ID":"90f8e17e-7c2a-4415-b101-a8357bd0ac33","Type":"ContainerStarted","Data":"fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850"}
Sep 16 02:30:54.821223 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.821205    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:30:54.836990 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.836946    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms" podStartSLOduration=1.491559142 podStartE2EDuration="1.836932193s" totalImagesPullingTime="345.373051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:54.835034126 +0000 UTC m=+754.265095514" watchObservedRunningTime="2026-09-16 02:30:54.836932193 +0000 UTC m=+754.266993641"
Sep 16 02:30:54.851150 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:54.851108    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q" podStartSLOduration=2.483938028 podStartE2EDuration="2.851092701s" totalImagesPullingTime="367.154673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:30:54.849382898 +0000 UTC m=+754.279444285" watchObservedRunningTime="2026-09-16 02:30:54.851092701 +0000 UTC m=+754.281154086"
Sep 16 02:30:58.818776 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:30:58.818748    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:31:00.831759 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:00.831727    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:31:00.832203 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:00.831796    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:31:12.107701 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:12.107664    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:31:20.710465 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.710417    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"]
Sep 16 02:31:20.715148 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.715122    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.717562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.717533    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-da6c4f-serving-cert\""
Sep 16 02:31:20.726613 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.726564    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"]
Sep 16 02:31:20.771749 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.771721    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.771885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.771772    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-maas-api-tls\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.771885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.771799    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t624\" (UniqueName: \"kubernetes.io/projected/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-kube-api-access-2t624\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.771885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.771825    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-metrics-certs\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.872932 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.872898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.873066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.872943    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-maas-api-tls\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.873066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.873053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t624\" (UniqueName: \"kubernetes.io/projected/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-kube-api-access-2t624\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.873192 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.873099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-metrics-certs\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.873672 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.873652    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.875280 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.875248    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-maas-api-tls\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.875476 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.875457    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-metrics-certs\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:20.884872 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:20.884851    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t624\" (UniqueName: \"kubernetes.io/projected/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-kube-api-access-2t624\") pod \"maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:21.031395 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:21.031315    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:21.306492 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:31:21.306460    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc9b2b1_9bdb_459b_8c4e_16265ac0d01b.slice/crio-47b99fc594f9d6ac80fe0bcb2853e16459f26d17c78beead45aa08c8344c5835 WatchSource:0}: Error finding container 47b99fc594f9d6ac80fe0bcb2853e16459f26d17c78beead45aa08c8344c5835: Status 404 returned error can't find the container with id 47b99fc594f9d6ac80fe0bcb2853e16459f26d17c78beead45aa08c8344c5835
Sep 16 02:31:21.308278 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:21.308254    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"]
Sep 16 02:31:21.963383 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:21.963342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb" event={"ID":"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b","Type":"ContainerStarted","Data":"e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460"}
Sep 16 02:31:21.963811 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:21.963398    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb" event={"ID":"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b","Type":"ContainerStarted","Data":"47b99fc594f9d6ac80fe0bcb2853e16459f26d17c78beead45aa08c8344c5835"}
Sep 16 02:31:21.963811 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:21.963543    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:21.980114 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:21.980069    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb" podStartSLOduration=1.707980837 podStartE2EDuration="1.980053434s" totalImagesPullingTime="272.072597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:21.978375772 +0000 UTC m=+781.408437158" watchObservedRunningTime="2026-09-16 02:31:21.980053434 +0000 UTC m=+781.410114848"
Sep 16 02:31:27.973579 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:27.973544    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:31:30.154241 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.154199    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"]
Sep 16 02:31:30.158988 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.158968    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.162601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.162573    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"default-dockercfg-w688b\""
Sep 16 02:31:30.162601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.162593    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"openshift-service-ca.crt\""
Sep 16 02:31:30.162888 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.162869    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"simulator-w0-5b59aa-kserve-self-signed-certs\""
Sep 16 02:31:30.162960 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.162898    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"kube-root-ca.crt\""
Sep 16 02:31:30.165286 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.165261    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"]
Sep 16 02:31:30.255268 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.255230    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-dshm\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.255452 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.255271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-model-cache\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.255452 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.255368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-home\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.255452 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.255449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tls-certs\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.255601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.255492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tmp-dir\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.255601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.255515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvsp\" (UniqueName: \"kubernetes.io/projected/55adf607-21aa-40ef-b1e6-8e8ba8232c85-kube-api-access-2qvsp\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.356585 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.356545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-dshm\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.356585 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.356586    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-model-cache\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.356817 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.356608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-home\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.356817 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.356627    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tls-certs\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.356817 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.356762    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tmp-dir\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.356817 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.356802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvsp\" (UniqueName: \"kubernetes.io/projected/55adf607-21aa-40ef-b1e6-8e8ba8232c85-kube-api-access-2qvsp\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.357067 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.357031    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-home\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.357067 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.357046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-model-cache\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.357204 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.357184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tmp-dir\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.358916 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.358893    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-dshm\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.359173 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.359156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tls-certs\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.364011 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.363993    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvsp\" (UniqueName: \"kubernetes.io/projected/55adf607-21aa-40ef-b1e6-8e8ba8232c85-kube-api-access-2qvsp\") pod \"simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") " pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.470584 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.470535    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:30.745785 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:30.745755    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"]
Sep 16 02:31:30.747762 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:31:30.747731    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55adf607_21aa_40ef_b1e6_8e8ba8232c85.slice/crio-89d0dd568111d6ac0249cadfb141e0dc4df7179d55586852167a37c55a2816ba WatchSource:0}: Error finding container 89d0dd568111d6ac0249cadfb141e0dc4df7179d55586852167a37c55a2816ba: Status 404 returned error can't find the container with id 89d0dd568111d6ac0249cadfb141e0dc4df7179d55586852167a37c55a2816ba
Sep 16 02:31:31.013775 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:31.013691    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb" event={"ID":"55adf607-21aa-40ef-b1e6-8e8ba8232c85","Type":"ContainerStarted","Data":"67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f"}
Sep 16 02:31:31.013775 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:31.013734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb" event={"ID":"55adf607-21aa-40ef-b1e6-8e8ba8232c85","Type":"ContainerStarted","Data":"89d0dd568111d6ac0249cadfb141e0dc4df7179d55586852167a37c55a2816ba"}
Sep 16 02:31:40.471026 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:40.470983    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:40.471026 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:40.471038    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:40.483636 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:40.483615    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:31:40.499773 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:40.499724    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb" podStartSLOduration=10.499706255 podStartE2EDuration="10.499706255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:31:31.029249377 +0000 UTC m=+790.459310764" watchObservedRunningTime="2026-09-16 02:31:40.499706255 +0000 UTC m=+799.929767645"
Sep 16 02:31:41.076559 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:31:41.076519    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:32:12.752190 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.752151    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"]
Sep 16 02:32:12.756832 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.756809    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.759116 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.759087    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"openshift-service-ca.crt\""
Sep 16 02:32:12.760408 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.760254    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"premium-w3-a3d7eb-kserve-self-signed-certs\""
Sep 16 02:32:12.760408 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.760257    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"default-dockercfg-klnxc\""
Sep 16 02:32:12.760408 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.760308    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a3d7eb\"/\"kube-root-ca.crt\""
Sep 16 02:32:12.763422 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.763395    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"]
Sep 16 02:32:12.842533 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.842492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4tn\" (UniqueName: \"kubernetes.io/projected/9514d97e-2329-4757-aa9c-eb06ca82b309-kube-api-access-jj4tn\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.842533 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.842537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-tmp-dir\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.842742 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.842643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-model-cache\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.842742 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.842684    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-dshm\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.842742 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.842700    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9514d97e-2329-4757-aa9c-eb06ca82b309-tls-certs\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.842742 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.842736    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-home\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.943962 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.943933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-model-cache\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944155 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.943980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-dshm\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944155 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.944008    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9514d97e-2329-4757-aa9c-eb06ca82b309-tls-certs\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944155 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.944031    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-home\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944155 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.944076    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4tn\" (UniqueName: \"kubernetes.io/projected/9514d97e-2329-4757-aa9c-eb06ca82b309-kube-api-access-jj4tn\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944155 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.944110    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-tmp-dir\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944463 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.944377    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-model-cache\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944536 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.944500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-home\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.944592 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.944538    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-tmp-dir\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.946236 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.946212    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-dshm\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.946581 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.946562    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9514d97e-2329-4757-aa9c-eb06ca82b309-tls-certs\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:12.951213 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:12.951194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4tn\" (UniqueName: \"kubernetes.io/projected/9514d97e-2329-4757-aa9c-eb06ca82b309-kube-api-access-jj4tn\") pod \"premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") " pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:13.068222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:13.068141    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:13.341697 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:13.341628    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"]
Sep 16 02:32:13.344064 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:32:13.344036    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9514d97e_2329_4757_aa9c_eb06ca82b309.slice/crio-42fc317ec9264f820408c6bf3c5f458a52ec8cdb8d27fb6f176c921177633657 WatchSource:0}: Error finding container 42fc317ec9264f820408c6bf3c5f458a52ec8cdb8d27fb6f176c921177633657: Status 404 returned error can't find the container with id 42fc317ec9264f820408c6bf3c5f458a52ec8cdb8d27fb6f176c921177633657
Sep 16 02:32:14.231677 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:14.231636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz" event={"ID":"9514d97e-2329-4757-aa9c-eb06ca82b309","Type":"ContainerStarted","Data":"e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d"}
Sep 16 02:32:14.232048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:14.231688    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz" event={"ID":"9514d97e-2329-4757-aa9c-eb06ca82b309","Type":"ContainerStarted","Data":"42fc317ec9264f820408c6bf3c5f458a52ec8cdb8d27fb6f176c921177633657"}
Sep 16 02:32:14.253614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:14.253567    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz" podStartSLOduration=2.253552747 podStartE2EDuration="2.253552747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:32:14.252090465 +0000 UTC m=+833.682151850" watchObservedRunningTime="2026-09-16 02:32:14.253552747 +0000 UTC m=+833.683614185"
Sep 16 02:32:23.068636 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:23.068602    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:23.068636 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:23.068646    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:23.081309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:23.081285    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:23.290959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:23.290925    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:32:36.107803 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:32:36.107765    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:33:20.994127 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:20.994044    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:33:20.996058 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:20.996031    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:33:23.551663 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.551628    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"]
Sep 16 02:33:23.555580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.555558    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.557915 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.557893    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da6c4f\"/\"kube-root-ca.crt\""
Sep 16 02:33:23.558886 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.558864    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-da6c4f\"/\"openshift-service-ca.crt\""
Sep 16 02:33:23.558886 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.558876    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da6c4f\"/\"default-dockercfg-vkjb6\""
Sep 16 02:33:23.559266 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.559244    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-da6c4f\"/\"test-model-da6c4f-kserve-self-signed-certs\""
Sep 16 02:33:23.562821 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.562798    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"]
Sep 16 02:33:23.588630 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.588608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-dshm\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.588733 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.588645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f08908-4fd7-4b41-920f-12279229a31a-tls-certs\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.588733 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.588671    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-home\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.588859 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.588761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vll7s\" (UniqueName: \"kubernetes.io/projected/10f08908-4fd7-4b41-920f-12279229a31a-kube-api-access-vll7s\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.588859 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.588802    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-model-cache\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.588859 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.588825    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-tmp-dir\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690043 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690008    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-dshm\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690043 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f08908-4fd7-4b41-920f-12279229a31a-tls-certs\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690299 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-home\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690299 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vll7s\" (UniqueName: \"kubernetes.io/projected/10f08908-4fd7-4b41-920f-12279229a31a-kube-api-access-vll7s\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690299 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690123    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-model-cache\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690299 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690140    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-tmp-dir\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690550 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-home\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690594 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690576    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-tmp-dir\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.690640 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.690624    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-model-cache\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.692292 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.692271    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-dshm\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.692508 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.692492    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f08908-4fd7-4b41-920f-12279229a31a-tls-certs\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.697580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.697557    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vll7s\" (UniqueName: \"kubernetes.io/projected/10f08908-4fd7-4b41-920f-12279229a31a-kube-api-access-vll7s\") pod \"test-model-da6c4f-kserve-7b9765f66f-vmtzn\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") " pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:23.867780 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:23.867704    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:24.140347 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:24.140323    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"]
Sep 16 02:33:24.142065 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:33:24.142041    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f08908_4fd7_4b41_920f_12279229a31a.slice/crio-7ddd3683a86222b5019b557a842f0c3587d2da9d20c563e6de324c3b8314d6be WatchSource:0}: Error finding container 7ddd3683a86222b5019b557a842f0c3587d2da9d20c563e6de324c3b8314d6be: Status 404 returned error can't find the container with id 7ddd3683a86222b5019b557a842f0c3587d2da9d20c563e6de324c3b8314d6be
Sep 16 02:33:24.600778 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:24.600738    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn" event={"ID":"10f08908-4fd7-4b41-920f-12279229a31a","Type":"ContainerStarted","Data":"d22cccf3d12783276359deea4bed64af1b0c63f24b8f4ed50f094a00cb630328"}
Sep 16 02:33:24.601184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:24.600792    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn" event={"ID":"10f08908-4fd7-4b41-920f-12279229a31a","Type":"ContainerStarted","Data":"7ddd3683a86222b5019b557a842f0c3587d2da9d20c563e6de324c3b8314d6be"}
Sep 16 02:33:24.616242 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:24.616180    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn" podStartSLOduration=1.616165175 podStartE2EDuration="1.616165175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:33:24.614983431 +0000 UTC m=+904.045044818" watchObservedRunningTime="2026-09-16 02:33:24.616165175 +0000 UTC m=+904.046226558"
Sep 16 02:33:33.868721 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:33.868684    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:33.869178 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:33.868751    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:33.881368 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:33.881344    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:34.664127 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:34.664100    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:33:53.137785 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.137755    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9097-121bb3"]
Sep 16 02:33:53.141441 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.141408    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:33:53.143823 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.143798    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7nx29\""
Sep 16 02:33:53.146988 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.146965    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9097-121bb3"]
Sep 16 02:33:53.169647 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.169621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99sv\" (UniqueName: \"kubernetes.io/projected/acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056-kube-api-access-p99sv\") pod \"test-curl-9097-121bb3\" (UID: \"acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056\") " pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:33:53.270902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.270871    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99sv\" (UniqueName: \"kubernetes.io/projected/acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056-kube-api-access-p99sv\") pod \"test-curl-9097-121bb3\" (UID: \"acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056\") " pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:33:53.277987 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.277965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99sv\" (UniqueName: \"kubernetes.io/projected/acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056-kube-api-access-p99sv\") pod \"test-curl-9097-121bb3\" (UID: \"acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056\") " pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:33:53.453041 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.453014    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:33:53.725086 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.722315    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9097-121bb3"]
Sep 16 02:33:53.725798 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:33:53.725772    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacc247d3_d7b0_4c1c_8bba_6f1fe8c0e056.slice/crio-b4ad828421c793542da15f7fdb6a2521c42edc59913a97a03284da1d1259bfae WatchSource:0}: Error finding container b4ad828421c793542da15f7fdb6a2521c42edc59913a97a03284da1d1259bfae: Status 404 returned error can't find the container with id b4ad828421c793542da15f7fdb6a2521c42edc59913a97a03284da1d1259bfae
Sep 16 02:33:53.752164 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:53.752140    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9097-121bb3" event={"ID":"acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056","Type":"ContainerStarted","Data":"b4ad828421c793542da15f7fdb6a2521c42edc59913a97a03284da1d1259bfae"}
Sep 16 02:33:56.769299 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:56.769260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9097-121bb3" event={"ID":"acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056","Type":"ContainerStarted","Data":"0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98"}
Sep 16 02:33:56.782341 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:56.782297    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9097-121bb3" podStartSLOduration=1.534462579 podStartE2EDuration="3.782281705s" totalImagesPullingTime="2.247819126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:33:56.781367128 +0000 UTC m=+936.211428511" watchObservedRunningTime="2026-09-16 02:33:56.782281705 +0000 UTC m=+936.212343090"
Sep 16 02:33:57.342117 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.342081    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9097-121bb3"]
Sep 16 02:33:57.347499 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.347476    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9097-121bb3"]
Sep 16 02:33:57.470468 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.470221    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9097-b02add"]
Sep 16 02:33:57.473460 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.471465    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" containerName="test-curl-9097-121bb3"
Sep 16 02:33:57.473460 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.471494    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" containerName="test-curl-9097-121bb3"
Sep 16 02:33:57.473460 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.471692    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" containerName="test-curl-9097-121bb3"
Sep 16 02:33:57.476078 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.476052    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:33:57.478275 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.478246    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9097-b02add"]
Sep 16 02:33:57.503805 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.503768    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842pt\" (UniqueName: \"kubernetes.io/projected/bb456292-47cd-4fad-9e96-77cccbd1b780-kube-api-access-842pt\") pod \"test-curl-9097-b02add\" (UID: \"bb456292-47cd-4fad-9e96-77cccbd1b780\") " pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:33:57.604272 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.604234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842pt\" (UniqueName: \"kubernetes.io/projected/bb456292-47cd-4fad-9e96-77cccbd1b780-kube-api-access-842pt\") pod \"test-curl-9097-b02add\" (UID: \"bb456292-47cd-4fad-9e96-77cccbd1b780\") " pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:33:57.611349 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.611324    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-842pt\" (UniqueName: \"kubernetes.io/projected/bb456292-47cd-4fad-9e96-77cccbd1b780-kube-api-access-842pt\") pod \"test-curl-9097-b02add\" (UID: \"bb456292-47cd-4fad-9e96-77cccbd1b780\") " pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:33:57.774787 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:33:57.774714    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9097-121bb3" container="test-curl-9097-121bb3"
Sep 16 02:33:57.787063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:57.787045    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:33:58.056218 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:58.056141    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9097-b02add"]
Sep 16 02:33:58.058494 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:33:58.058465    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb456292_47cd_4fad_9e96_77cccbd1b780.slice/crio-6646b469561a7d768b3e228caf5f06a954117398f302ac229314f2f602565a15 WatchSource:0}: Error finding container 6646b469561a7d768b3e228caf5f06a954117398f302ac229314f2f602565a15: Status 404 returned error can't find the container with id 6646b469561a7d768b3e228caf5f06a954117398f302ac229314f2f602565a15
Sep 16 02:33:58.781345 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:58.781313    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9097-b02add" event={"ID":"bb456292-47cd-4fad-9e96-77cccbd1b780","Type":"ContainerStarted","Data":"55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14"}
Sep 16 02:33:58.783746 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:58.781354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9097-b02add" event={"ID":"bb456292-47cd-4fad-9e96-77cccbd1b780","Type":"ContainerStarted","Data":"6646b469561a7d768b3e228caf5f06a954117398f302ac229314f2f602565a15"}
Sep 16 02:33:58.783746 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:58.781585    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9097-121bb3" podUID="acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" containerName="test-curl-9097-121bb3" containerID="cri-o://0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98" gracePeriod=2
Sep 16 02:33:58.795688 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:58.795643    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9097-b02add" podStartSLOduration=1.795628416 podStartE2EDuration="1.795628416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:33:58.793865241 +0000 UTC m=+938.223926629" watchObservedRunningTime="2026-09-16 02:33:58.795628416 +0000 UTC m=+938.225689837"
Sep 16 02:33:58.795824 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:33:58.795707    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9097-121bb3\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:33:59.354958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:59.354876    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9097-b02add"]
Sep 16 02:33:59.359192 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:33:59.359162    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9097-b02add"]
Sep 16 02:33:59.786092 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:33:59.786063    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9097-b02add" container="test-curl-9097-b02add"
Sep 16 02:34:00.791937 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:00.791905    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9097-b02add" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" containerName="test-curl-9097-b02add" containerID="cri-o://55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14" gracePeriod=2
Sep 16 02:34:00.794468 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:34:00.794421    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9097-b02add\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:34:00.972729 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:00.972707    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:34:00.975263 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:34:00.975239    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9097-b02add\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:34:01.037688 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.037628    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p99sv\" (UniqueName: \"kubernetes.io/projected/acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056-kube-api-access-p99sv\") pod \"acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056\" (UID: \"acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056\") "
Sep 16 02:34:01.039561 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.039541    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056-kube-api-access-p99sv" (OuterVolumeSpecName: "kube-api-access-p99sv") pod "acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" (UID: "acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056"). InnerVolumeSpecName "kube-api-access-p99sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:34:01.113030 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:34:01.112993    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9097-b02add\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:34:01.113949 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.113930    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" path="/var/lib/kubelet/pods/acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056/volumes"
Sep 16 02:34:01.138342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.138319    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p99sv\" (UniqueName: \"kubernetes.io/projected/acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056-kube-api-access-p99sv\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:34:01.798786 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.798753    2556 generic.go:356] "Generic (PLEG): container finished" podID="acc247d3-d7b0-4c1c-8bba-6f1fe8c0e056" containerID="0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98" exitCode=137
Sep 16 02:34:01.799269 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.798805    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-121bb3"
Sep 16 02:34:01.799269 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.798857    2556 scope.go:117] "RemoveContainer" containerID="0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98"
Sep 16 02:34:01.809080 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.809063    2556 scope.go:117] "RemoveContainer" containerID="0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98"
Sep 16 02:34:01.809338 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:34:01.809316    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98\": container with ID starting with 0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98 not found: ID does not exist" containerID="0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98"
Sep 16 02:34:01.809400 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:01.809346    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98"} err="failed to get container status \"0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98\": rpc error: code = NotFound desc = could not find container \"0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98\": container with ID starting with 0a51fdc3c9da1b4454a849fc8bb76131a14977c720777e731af3e08e7e9d5c98 not found: ID does not exist"
Sep 16 02:34:02.975738 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:02.975714    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:34:03.058126 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.058041    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-842pt\" (UniqueName: \"kubernetes.io/projected/bb456292-47cd-4fad-9e96-77cccbd1b780-kube-api-access-842pt\") pod \"bb456292-47cd-4fad-9e96-77cccbd1b780\" (UID: \"bb456292-47cd-4fad-9e96-77cccbd1b780\") "
Sep 16 02:34:03.060017 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.059992    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb456292-47cd-4fad-9e96-77cccbd1b780-kube-api-access-842pt" (OuterVolumeSpecName: "kube-api-access-842pt") pod "bb456292-47cd-4fad-9e96-77cccbd1b780" (UID: "bb456292-47cd-4fad-9e96-77cccbd1b780"). InnerVolumeSpecName "kube-api-access-842pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:34:03.114016 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.113977    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" path="/var/lib/kubelet/pods/bb456292-47cd-4fad-9e96-77cccbd1b780/volumes"
Sep 16 02:34:03.159003 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.158973    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842pt\" (UniqueName: \"kubernetes.io/projected/bb456292-47cd-4fad-9e96-77cccbd1b780-kube-api-access-842pt\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:34:03.812611 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.812575    2556 generic.go:356] "Generic (PLEG): container finished" podID="bb456292-47cd-4fad-9e96-77cccbd1b780" containerID="55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14" exitCode=137
Sep 16 02:34:03.812800 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.812617    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9097-b02add"
Sep 16 02:34:03.812800 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.812632    2556 scope.go:117] "RemoveContainer" containerID="55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14"
Sep 16 02:34:03.821614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.821594    2556 scope.go:117] "RemoveContainer" containerID="55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14"
Sep 16 02:34:03.821909 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:34:03.821886    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14\": container with ID starting with 55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14 not found: ID does not exist" containerID="55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14"
Sep 16 02:34:03.821965 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:03.821919    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14"} err="failed to get container status \"55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14\": rpc error: code = NotFound desc = could not find container \"55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14\": container with ID starting with 55b10c8564b24718709ad42bd74eeae6f76ea4a8229f4c5fac6683ae4f84bb14 not found: ID does not exist"
Sep 16 02:34:04.107881 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:04.107807    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:34:39.245398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.245361    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"]
Sep 16 02:34:39.245961 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.245804    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" containerName="test-curl-9097-b02add"
Sep 16 02:34:39.245961 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.245817    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" containerName="test-curl-9097-b02add"
Sep 16 02:34:39.245961 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.245891    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb456292-47cd-4fad-9e96-77cccbd1b780" containerName="test-curl-9097-b02add"
Sep 16 02:34:39.257360 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.257328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.259978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.259948    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"]
Sep 16 02:34:39.260170 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.259984    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5b59aa\"/\"distinct-2-w0-5b59aa-kserve-self-signed-certs\""
Sep 16 02:34:39.295202 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.295170    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-dshm\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.295340 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.295236    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-home\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.295340 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.295312    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfnkx\" (UniqueName: \"kubernetes.io/projected/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-kube-api-access-qfnkx\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.295423 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.295347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-model-cache\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.295423 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.295382    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tmp-dir\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.295543 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.295472    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tls-certs\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.396699 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.396656    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-home\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.396890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.396736    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnkx\" (UniqueName: \"kubernetes.io/projected/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-kube-api-access-qfnkx\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.396890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.396772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-model-cache\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.396890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.396802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tmp-dir\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.396890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.396842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tls-certs\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.396890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.396884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-dshm\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.397234 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.397210    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tmp-dir\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.397330 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.397312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-home\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.397669 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.397645    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-model-cache\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.399848 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.399824    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-dshm\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.399940 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.399897    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tls-certs\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.406947 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.404025    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnkx\" (UniqueName: \"kubernetes.io/projected/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-kube-api-access-qfnkx\") pod \"distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") " pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.569234 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.569148    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:39.840606 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:39.840581    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"]
Sep 16 02:34:39.842342 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:34:39.842315    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ebcd016_962e_45c6_a0a9_5d1d909cb1cb.slice/crio-7b0e1324828cf850c162a194df1b3bb57d7d682d52e17346585c52a6a396fc4f WatchSource:0}: Error finding container 7b0e1324828cf850c162a194df1b3bb57d7d682d52e17346585c52a6a396fc4f: Status 404 returned error can't find the container with id 7b0e1324828cf850c162a194df1b3bb57d7d682d52e17346585c52a6a396fc4f
Sep 16 02:34:40.004066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:40.004027    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4" event={"ID":"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb","Type":"ContainerStarted","Data":"6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2"}
Sep 16 02:34:40.004066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:40.004071    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4" event={"ID":"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb","Type":"ContainerStarted","Data":"7b0e1324828cf850c162a194df1b3bb57d7d682d52e17346585c52a6a396fc4f"}
Sep 16 02:34:40.019013 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:40.018965    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4" podStartSLOduration=1.018948659 podStartE2EDuration="1.018948659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:34:40.017561403 +0000 UTC m=+979.447622808" watchObservedRunningTime="2026-09-16 02:34:40.018948659 +0000 UTC m=+979.449010067"
Sep 16 02:34:49.569524 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:49.569408    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:49.569524 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:49.569489    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:49.582057 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:49.582031    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:50.065759 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:50.065729    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:34:58.496621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.496331    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"]
Sep 16 02:34:58.502316 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.502294    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.504872 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.504793    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-06e75986-serving-cert\""
Sep 16 02:34:58.508234 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.508201    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"]
Sep 16 02:34:58.591983 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.591951    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-maas-api-tls\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.592171 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.592009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-openshift-service-ca\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.592171 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.592071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-metrics-certs\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.592171 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.592149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bqh\" (UniqueName: \"kubernetes.io/projected/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-kube-api-access-76bqh\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.693261 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.693230    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-maas-api-tls\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.693402 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.693275    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-openshift-service-ca\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.693402 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.693300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-metrics-certs\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.693402 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.693357    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bqh\" (UniqueName: \"kubernetes.io/projected/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-kube-api-access-76bqh\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.693956 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.693931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-openshift-service-ca\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.695899 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.695859    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-maas-api-tls\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.696000 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.695965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-metrics-certs\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.701288 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.701264    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bqh\" (UniqueName: \"kubernetes.io/projected/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-kube-api-access-76bqh\") pod \"maas-api-e2e-ait-06e75986-b99f698d8-2pqhx\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:58.816809 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:58.816724    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:34:59.092595 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:59.092570    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"]
Sep 16 02:34:59.094181 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:34:59.094148    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b7e060_a42b_4bb7_ae85_9c0d1a8de31f.slice/crio-bf3747c9faf965194b620affbfbc40eec385bb419d27b4ab216d01c63609c9d5 WatchSource:0}: Error finding container bf3747c9faf965194b620affbfbc40eec385bb419d27b4ab216d01c63609c9d5: Status 404 returned error can't find the container with id bf3747c9faf965194b620affbfbc40eec385bb419d27b4ab216d01c63609c9d5
Sep 16 02:34:59.095716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:59.095694    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:34:59.104370 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:34:59.104286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx" event={"ID":"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f","Type":"ContainerStarted","Data":"bf3747c9faf965194b620affbfbc40eec385bb419d27b4ab216d01c63609c9d5"}
Sep 16 02:35:00.112800 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:00.112763    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx" event={"ID":"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f","Type":"ContainerStarted","Data":"cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b"}
Sep 16 02:35:00.113184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:00.112838    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:35:00.128797 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:00.128748    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx" podStartSLOduration=1.592230896 podStartE2EDuration="2.128732828s" totalImagesPullingTime="536.501932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:00.127888278 +0000 UTC m=+999.557949662" watchObservedRunningTime="2026-09-16 02:35:00.128732828 +0000 UTC m=+999.558794213"
Sep 16 02:35:05.035786 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:05.035753    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"]
Sep 16 02:35:06.123774 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:06.123745    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:35:13.108628 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:13.108595    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:35:15.897671 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:15.897634    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"]
Sep 16 02:35:15.898154 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:15.897871    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b" podUID="2632151a-ead4-4d89-a482-f788058d0545" containerName="maas-api" containerID="cri-o://e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488" gracePeriod=30
Sep 16 02:35:16.154607 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.154547    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:35:16.163103 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.163080    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m65w8\" (UniqueName: \"kubernetes.io/projected/2632151a-ead4-4d89-a482-f788058d0545-kube-api-access-m65w8\") pod \"2632151a-ead4-4d89-a482-f788058d0545\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") "
Sep 16 02:35:16.163215 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.163108    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-maas-api-tls\") pod \"2632151a-ead4-4d89-a482-f788058d0545\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") "
Sep 16 02:35:16.163215 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.163130    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2632151a-ead4-4d89-a482-f788058d0545-openshift-service-ca\") pod \"2632151a-ead4-4d89-a482-f788058d0545\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") "
Sep 16 02:35:16.163307 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.163271    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-metrics-certs\") pod \"2632151a-ead4-4d89-a482-f788058d0545\" (UID: \"2632151a-ead4-4d89-a482-f788058d0545\") "
Sep 16 02:35:16.163906 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.163863    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2632151a-ead4-4d89-a482-f788058d0545-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2632151a-ead4-4d89-a482-f788058d0545" (UID: "2632151a-ead4-4d89-a482-f788058d0545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:16.164416 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.164391    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2632151a-ead4-4d89-a482-f788058d0545-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:16.166031 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.165907    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2632151a-ead4-4d89-a482-f788058d0545-kube-api-access-m65w8" (OuterVolumeSpecName: "kube-api-access-m65w8") pod "2632151a-ead4-4d89-a482-f788058d0545" (UID: "2632151a-ead4-4d89-a482-f788058d0545"). InnerVolumeSpecName "kube-api-access-m65w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:16.166578 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.166543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2632151a-ead4-4d89-a482-f788058d0545" (UID: "2632151a-ead4-4d89-a482-f788058d0545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:16.173638 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.173603    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2632151a-ead4-4d89-a482-f788058d0545" (UID: "2632151a-ead4-4d89-a482-f788058d0545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:16.206382 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.206347    2556 generic.go:356] "Generic (PLEG): container finished" podID="2632151a-ead4-4d89-a482-f788058d0545" containerID="e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488" exitCode=0
Sep 16 02:35:16.206559 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.206402    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"
Sep 16 02:35:16.206559 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.206461    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b" event={"ID":"2632151a-ead4-4d89-a482-f788058d0545","Type":"ContainerDied","Data":"e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488"}
Sep 16 02:35:16.206559 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.206547    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b" event={"ID":"2632151a-ead4-4d89-a482-f788058d0545","Type":"ContainerDied","Data":"4fb70bad6776ba8b0d5af3a13011bc564f8b597ce7ab841fdc9d73b1111e4902"}
Sep 16 02:35:16.206705 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.206579    2556 scope.go:117] "RemoveContainer" containerID="e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488"
Sep 16 02:35:16.217317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.217284    2556 scope.go:117] "RemoveContainer" containerID="e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488"
Sep 16 02:35:16.217663 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:16.217642    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488\": container with ID starting with e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488 not found: ID does not exist" containerID="e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488"
Sep 16 02:35:16.217760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.217672    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488"} err="failed to get container status \"e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488\": rpc error: code = NotFound desc = could not find container \"e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488\": container with ID starting with e78b8372154ed721bea84ba487bed6893cf30c91befa77ba65d3e3b544069488 not found: ID does not exist"
Sep 16 02:35:16.239613 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.239590    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"]
Sep 16 02:35:16.247508 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.247486    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dfb47-548ccf7456-6v44b"]
Sep 16 02:35:16.265924 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.265906    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:16.266008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.265926    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65w8\" (UniqueName: \"kubernetes.io/projected/2632151a-ead4-4d89-a482-f788058d0545-kube-api-access-m65w8\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:16.266008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:16.265936    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2632151a-ead4-4d89-a482-f788058d0545-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:17.114949 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:17.114917    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2632151a-ead4-4d89-a482-f788058d0545" path="/var/lib/kubelet/pods/2632151a-ead4-4d89-a482-f788058d0545/volumes"
Sep 16 02:35:20.055121 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.055056    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn" podUID="10f08908-4fd7-4b41-920f-12279229a31a" containerName="main" containerID="cri-o://d22cccf3d12783276359deea4bed64af1b0c63f24b8f4ed50f094a00cb630328" gracePeriod=45
Sep 16 02:35:20.231041 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.231012    2556 generic.go:356] "Generic (PLEG): container finished" podID="10f08908-4fd7-4b41-920f-12279229a31a" containerID="d22cccf3d12783276359deea4bed64af1b0c63f24b8f4ed50f094a00cb630328" exitCode=0
Sep 16 02:35:20.231164 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.231078    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn" event={"ID":"10f08908-4fd7-4b41-920f-12279229a31a","Type":"ContainerDied","Data":"d22cccf3d12783276359deea4bed64af1b0c63f24b8f4ed50f094a00cb630328"}
Sep 16 02:35:20.312196 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.312134    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:35:20.400287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400254    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-model-cache\") pod \"10f08908-4fd7-4b41-920f-12279229a31a\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") "
Sep 16 02:35:20.400475 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400306    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-tmp-dir\") pod \"10f08908-4fd7-4b41-920f-12279229a31a\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") "
Sep 16 02:35:20.400475 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400344    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f08908-4fd7-4b41-920f-12279229a31a-tls-certs\") pod \"10f08908-4fd7-4b41-920f-12279229a31a\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") "
Sep 16 02:35:20.400475 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400373    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-home\") pod \"10f08908-4fd7-4b41-920f-12279229a31a\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") "
Sep 16 02:35:20.400475 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400403    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-dshm\") pod \"10f08908-4fd7-4b41-920f-12279229a31a\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") "
Sep 16 02:35:20.400475 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400463    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vll7s\" (UniqueName: \"kubernetes.io/projected/10f08908-4fd7-4b41-920f-12279229a31a-kube-api-access-vll7s\") pod \"10f08908-4fd7-4b41-920f-12279229a31a\" (UID: \"10f08908-4fd7-4b41-920f-12279229a31a\") "
Sep 16 02:35:20.400750 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400539    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-model-cache" (OuterVolumeSpecName: "model-cache") pod "10f08908-4fd7-4b41-920f-12279229a31a" (UID: "10f08908-4fd7-4b41-920f-12279229a31a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:20.400750 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400641    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10f08908-4fd7-4b41-920f-12279229a31a" (UID: "10f08908-4fd7-4b41-920f-12279229a31a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:20.400750 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400733    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-home" (OuterVolumeSpecName: "home") pod "10f08908-4fd7-4b41-920f-12279229a31a" (UID: "10f08908-4fd7-4b41-920f-12279229a31a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:20.400977 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400887    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:20.400977 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400906    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:20.400977 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.400917    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:20.402622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.402599    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-dshm" (OuterVolumeSpecName: "dshm") pod "10f08908-4fd7-4b41-920f-12279229a31a" (UID: "10f08908-4fd7-4b41-920f-12279229a31a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:35:20.402782 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.402768    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f08908-4fd7-4b41-920f-12279229a31a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10f08908-4fd7-4b41-920f-12279229a31a" (UID: "10f08908-4fd7-4b41-920f-12279229a31a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:20.402833 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.402785    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f08908-4fd7-4b41-920f-12279229a31a-kube-api-access-vll7s" (OuterVolumeSpecName: "kube-api-access-vll7s") pod "10f08908-4fd7-4b41-920f-12279229a31a" (UID: "10f08908-4fd7-4b41-920f-12279229a31a"). InnerVolumeSpecName "kube-api-access-vll7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:20.501610 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.501579    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f08908-4fd7-4b41-920f-12279229a31a-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:20.501610 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.501604    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f08908-4fd7-4b41-920f-12279229a31a-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:20.501610 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:20.501614    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vll7s\" (UniqueName: \"kubernetes.io/projected/10f08908-4fd7-4b41-920f-12279229a31a-kube-api-access-vll7s\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:21.237305 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.237268    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"
Sep 16 02:35:21.237784 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.237297    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn" event={"ID":"10f08908-4fd7-4b41-920f-12279229a31a","Type":"ContainerDied","Data":"7ddd3683a86222b5019b557a842f0c3587d2da9d20c563e6de324c3b8314d6be"}
Sep 16 02:35:21.237784 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.237355    2556 scope.go:117] "RemoveContainer" containerID="d22cccf3d12783276359deea4bed64af1b0c63f24b8f4ed50f094a00cb630328"
Sep 16 02:35:21.252174 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.252144    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"]
Sep 16 02:35:21.255679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.255657    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-da6c4f/test-model-da6c4f-kserve-7b9765f66f-vmtzn"]
Sep 16 02:35:21.664741 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.664711    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"]
Sep 16 02:35:21.664997 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.664955    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx" podUID="58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" containerName="maas-api" containerID="cri-o://cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b" gracePeriod=30
Sep 16 02:35:21.912794 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:21.912768    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:35:22.013738 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.013711    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-maas-api-tls\") pod \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") "
Sep 16 02:35:22.013924 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.013797    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-openshift-service-ca\") pod \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") "
Sep 16 02:35:22.013924 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.013876    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76bqh\" (UniqueName: \"kubernetes.io/projected/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-kube-api-access-76bqh\") pod \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") "
Sep 16 02:35:22.014079 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.013985    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-metrics-certs\") pod \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\" (UID: \"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f\") "
Sep 16 02:35:22.014207 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.014179    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" (UID: "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:22.014361 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.014338    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.017197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.017169    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" (UID: "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:22.018254 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.018229    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-kube-api-access-76bqh" (OuterVolumeSpecName: "kube-api-access-76bqh") pod "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" (UID: "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f"). InnerVolumeSpecName "kube-api-access-76bqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:22.018448 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.018249    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" (UID: "58b7e060-a42b-4bb7-ae85-9c0d1a8de31f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:22.115337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.115303    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76bqh\" (UniqueName: \"kubernetes.io/projected/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-kube-api-access-76bqh\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.115337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.115331    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.115337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.115341    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:22.244374 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.244335    2556 generic.go:356] "Generic (PLEG): container finished" podID="58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" containerID="cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b" exitCode=0
Sep 16 02:35:22.244806 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.244376    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"
Sep 16 02:35:22.244806 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.244387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx" event={"ID":"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f","Type":"ContainerDied","Data":"cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b"}
Sep 16 02:35:22.244806 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.244457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx" event={"ID":"58b7e060-a42b-4bb7-ae85-9c0d1a8de31f","Type":"ContainerDied","Data":"bf3747c9faf965194b620affbfbc40eec385bb419d27b4ab216d01c63609c9d5"}
Sep 16 02:35:22.244806 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.244484    2556 scope.go:117] "RemoveContainer" containerID="cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b"
Sep 16 02:35:22.253802 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.253785    2556 scope.go:117] "RemoveContainer" containerID="cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b"
Sep 16 02:35:22.254094 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:22.254073    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b\": container with ID starting with cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b not found: ID does not exist" containerID="cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b"
Sep 16 02:35:22.254165 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.254103    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b"} err="failed to get container status \"cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b\": rpc error: code = NotFound desc = could not find container \"cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b\": container with ID starting with cd1a76119d8a1b246fb0bc464609549cadaa58369bea6df0cc26a3f948299e0b not found: ID does not exist"
Sep 16 02:35:22.264553 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.264504    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"]
Sep 16 02:35:22.269594 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:22.269573    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-06e75986-b99f698d8-2pqhx"]
Sep 16 02:35:23.114112 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:23.114073    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f08908-4fd7-4b41-920f-12279229a31a" path="/var/lib/kubelet/pods/10f08908-4fd7-4b41-920f-12279229a31a/volumes"
Sep 16 02:35:23.114745 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:23.114719    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" path="/var/lib/kubelet/pods/58b7e060-a42b-4bb7-ae85-9c0d1a8de31f/volumes"
Sep 16 02:35:26.613576 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.613539    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"]
Sep 16 02:35:26.614159 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614136    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" containerName="maas-api"
Sep 16 02:35:26.614159 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614154    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" containerName="maas-api"
Sep 16 02:35:26.614265 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614191    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2632151a-ead4-4d89-a482-f788058d0545" containerName="maas-api"
Sep 16 02:35:26.614265 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614200    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2632151a-ead4-4d89-a482-f788058d0545" containerName="maas-api"
Sep 16 02:35:26.614265 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614214    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f08908-4fd7-4b41-920f-12279229a31a" containerName="main"
Sep 16 02:35:26.614265 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614223    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f08908-4fd7-4b41-920f-12279229a31a" containerName="main"
Sep 16 02:35:26.614380 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614316    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f08908-4fd7-4b41-920f-12279229a31a" containerName="main"
Sep 16 02:35:26.614380 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614328    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b7e060-a42b-4bb7-ae85-9c0d1a8de31f" containerName="maas-api"
Sep 16 02:35:26.614380 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.614339    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2632151a-ead4-4d89-a482-f788058d0545" containerName="maas-api"
Sep 16 02:35:26.618645 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.618625    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.620855 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.620831    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c40df7fa-serving-cert\""
Sep 16 02:35:26.625617 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.625529    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"]
Sep 16 02:35:26.656782 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.656744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbm95\" (UniqueName: \"kubernetes.io/projected/8052b8a0-4ac9-4ef4-b437-b62f088117e1-kube-api-access-qbm95\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.656922 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.656801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052b8a0-4ac9-4ef4-b437-b62f088117e1-openshift-service-ca\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.656922 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.656850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-metrics-certs\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.657033 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.656957    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.757831 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.757801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.757991 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:26.757962    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-serving-cert: secret "maas-api-e2e-migrate-c40df7fa-serving-cert" not found
Sep 16 02:35:26.758075 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:26.758062    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls podName:8052b8a0-4ac9-4ef4-b437-b62f088117e1 nodeName:}" failed. No retries permitted until 2026-09-16 02:35:27.258041376 +0000 UTC m=+1026.688102754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls") pod "maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl" (UID: "8052b8a0-4ac9-4ef4-b437-b62f088117e1") : secret "maas-api-e2e-migrate-c40df7fa-serving-cert" not found
Sep 16 02:35:26.758145 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.757967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbm95\" (UniqueName: \"kubernetes.io/projected/8052b8a0-4ac9-4ef4-b437-b62f088117e1-kube-api-access-qbm95\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.758145 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.758105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052b8a0-4ac9-4ef4-b437-b62f088117e1-openshift-service-ca\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.758145 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.758137    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-metrics-certs\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.758864 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.758834    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052b8a0-4ac9-4ef4-b437-b62f088117e1-openshift-service-ca\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.760561 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.760534    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-metrics-certs\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:26.765559 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:26.765540    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbm95\" (UniqueName: \"kubernetes.io/projected/8052b8a0-4ac9-4ef4-b437-b62f088117e1-kube-api-access-qbm95\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:27.262864 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:27.262832    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:27.265310 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:27.265284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls\") pod \"maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:27.533890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:27.533804    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:27.807476 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:27.807449    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"]
Sep 16 02:35:27.809644 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:35:27.809613    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8052b8a0_4ac9_4ef4_b437_b62f088117e1.slice/crio-deea9fe3f4c81fa788e31c195d9b8a0afaf05668b6ad843db8eb57bd6b25a0e6 WatchSource:0}: Error finding container deea9fe3f4c81fa788e31c195d9b8a0afaf05668b6ad843db8eb57bd6b25a0e6: Status 404 returned error can't find the container with id deea9fe3f4c81fa788e31c195d9b8a0afaf05668b6ad843db8eb57bd6b25a0e6
Sep 16 02:35:28.280035 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:28.280002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl" event={"ID":"8052b8a0-4ac9-4ef4-b437-b62f088117e1","Type":"ContainerStarted","Data":"deea9fe3f4c81fa788e31c195d9b8a0afaf05668b6ad843db8eb57bd6b25a0e6"}
Sep 16 02:35:29.286882 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:29.286847    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl" event={"ID":"8052b8a0-4ac9-4ef4-b437-b62f088117e1","Type":"ContainerStarted","Data":"64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6"}
Sep 16 02:35:29.287269 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:29.286934    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:29.301620 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:29.301562    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl" podStartSLOduration=2.851478524 podStartE2EDuration="3.301546069s" totalImagesPullingTime="450.067545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:29.30136944 +0000 UTC m=+1028.731430828" watchObservedRunningTime="2026-09-16 02:35:29.301546069 +0000 UTC m=+1028.731607456"
Sep 16 02:35:34.221638 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.221574    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"]
Sep 16 02:35:34.222226 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.221876    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb" podUID="4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" containerName="maas-api" containerID="cri-o://e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460" gracePeriod=30
Sep 16 02:35:34.690968 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.690943    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:35:34.832950 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.832865    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t624\" (UniqueName: \"kubernetes.io/projected/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-kube-api-access-2t624\") pod \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") "
Sep 16 02:35:34.832950 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.832931    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-metrics-certs\") pod \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") "
Sep 16 02:35:34.833180 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.832964    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-maas-api-tls\") pod \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") "
Sep 16 02:35:34.833180 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.833025    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-openshift-service-ca\") pod \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\" (UID: \"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b\") "
Sep 16 02:35:34.833447 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.833393    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" (UID: "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:34.834960 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.834932    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-kube-api-access-2t624" (OuterVolumeSpecName: "kube-api-access-2t624") pod "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" (UID: "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b"). InnerVolumeSpecName "kube-api-access-2t624". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:34.835085 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.834971    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" (UID: "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:34.835085 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.835054    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" (UID: "4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:34.934057 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.934019    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:34.934057 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.934053    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t624\" (UniqueName: \"kubernetes.io/projected/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-kube-api-access-2t624\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:34.934057 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.934064    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:34.934304 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:34.934073    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:35.297412 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.297378    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:35.322404 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.322288    2556 generic.go:356] "Generic (PLEG): container finished" podID="4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" containerID="e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460" exitCode=0
Sep 16 02:35:35.322404 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.322366    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"
Sep 16 02:35:35.322663 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.322422    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb" event={"ID":"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b","Type":"ContainerDied","Data":"e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460"}
Sep 16 02:35:35.322663 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.322493    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb" event={"ID":"4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b","Type":"ContainerDied","Data":"47b99fc594f9d6ac80fe0bcb2853e16459f26d17c78beead45aa08c8344c5835"}
Sep 16 02:35:35.322663 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.322521    2556 scope.go:117] "RemoveContainer" containerID="e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460"
Sep 16 02:35:35.333359 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.333332    2556 scope.go:117] "RemoveContainer" containerID="e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460"
Sep 16 02:35:35.333738 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:35.333716    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460\": container with ID starting with e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460 not found: ID does not exist" containerID="e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460"
Sep 16 02:35:35.333825 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.333751    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460"} err="failed to get container status \"e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460\": rpc error: code = NotFound desc = could not find container \"e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460\": container with ID starting with e966c197bd53474ea20742236ecd235c01e112ad45c4e867ce8454903ffcb460 not found: ID does not exist"
Sep 16 02:35:35.342413 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.342377    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"]
Sep 16 02:35:35.344611 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:35.344588    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-da6c4f-fbd49bfbb-p8wlb"]
Sep 16 02:35:37.113910 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:37.113880    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" path="/var/lib/kubelet/pods/4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b/volumes"
Sep 16 02:35:47.316537 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.316500    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"]
Sep 16 02:35:47.317147 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.317127    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" containerName="maas-api"
Sep 16 02:35:47.317216 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.317151    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" containerName="maas-api"
Sep 16 02:35:47.317300 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.317286    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc9b2b1-9bdb-459b-8c4e-16265ac0d01b" containerName="maas-api"
Sep 16 02:35:47.323735 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.323705    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.325346 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.325321    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"]
Sep 16 02:35:47.326964 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.326942    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-db1ed7-serving-cert\""
Sep 16 02:35:47.452819 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.452784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp77x\" (UniqueName: \"kubernetes.io/projected/26fd1472-b966-4d6e-8f5c-984005162c5a-kube-api-access-kp77x\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.452971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.452845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd1472-b966-4d6e-8f5c-984005162c5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.452971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.452881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.452971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.452955    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.554003 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.553971    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp77x\" (UniqueName: \"kubernetes.io/projected/26fd1472-b966-4d6e-8f5c-984005162c5a-kube-api-access-kp77x\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.554154 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.554031    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd1472-b966-4d6e-8f5c-984005162c5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.554154 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.554048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.554154 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.554101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.554655 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.554629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd1472-b966-4d6e-8f5c-984005162c5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.556389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.556367    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.556555 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.556417    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.568585 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.568526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp77x\" (UniqueName: \"kubernetes.io/projected/26fd1472-b966-4d6e-8f5c-984005162c5a-kube-api-access-kp77x\") pod \"maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.638649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.638615    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:47.914149 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:47.914125    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"]
Sep 16 02:35:47.918818 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:35:47.918781    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fd1472_b966_4d6e_8f5c_984005162c5a.slice/crio-a0410f6686458070b8b38032e42c8d3008f39c27671e44c810290289d46e2b8b WatchSource:0}: Error finding container a0410f6686458070b8b38032e42c8d3008f39c27671e44c810290289d46e2b8b: Status 404 returned error can't find the container with id a0410f6686458070b8b38032e42c8d3008f39c27671e44c810290289d46e2b8b
Sep 16 02:35:48.391637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:48.391609    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2" event={"ID":"26fd1472-b966-4d6e-8f5c-984005162c5a","Type":"ContainerStarted","Data":"a0410f6686458070b8b38032e42c8d3008f39c27671e44c810290289d46e2b8b"}
Sep 16 02:35:49.399667 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:49.399626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2" event={"ID":"26fd1472-b966-4d6e-8f5c-984005162c5a","Type":"ContainerStarted","Data":"f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f"}
Sep 16 02:35:49.400072 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:49.399815    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:49.417196 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:49.417147    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2" podStartSLOduration=1.969824938 podStartE2EDuration="2.417133736s" totalImagesPullingTime="447.308798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:49.414471492 +0000 UTC m=+1048.844532879" watchObservedRunningTime="2026-09-16 02:35:49.417133736 +0000 UTC m=+1048.847195162"
Sep 16 02:35:50.420100 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.420071    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"]
Sep 16 02:35:50.420544 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.420294    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl" podUID="8052b8a0-4ac9-4ef4-b437-b62f088117e1" containerName="maas-api" containerID="cri-o://64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6" gracePeriod=30
Sep 16 02:35:50.685724 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.685661    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:50.789691 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.789664    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-metrics-certs\") pod \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") "
Sep 16 02:35:50.789853 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.789740    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls\") pod \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") "
Sep 16 02:35:50.789853 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.789770    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052b8a0-4ac9-4ef4-b437-b62f088117e1-openshift-service-ca\") pod \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") "
Sep 16 02:35:50.789853 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.789840    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbm95\" (UniqueName: \"kubernetes.io/projected/8052b8a0-4ac9-4ef4-b437-b62f088117e1-kube-api-access-qbm95\") pod \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\" (UID: \"8052b8a0-4ac9-4ef4-b437-b62f088117e1\") "
Sep 16 02:35:50.790164 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.790130    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8052b8a0-4ac9-4ef4-b437-b62f088117e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8052b8a0-4ac9-4ef4-b437-b62f088117e1" (UID: "8052b8a0-4ac9-4ef4-b437-b62f088117e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:35:50.791863 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.791836    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8052b8a0-4ac9-4ef4-b437-b62f088117e1" (UID: "8052b8a0-4ac9-4ef4-b437-b62f088117e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:50.791957 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.791845    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8052b8a0-4ac9-4ef4-b437-b62f088117e1" (UID: "8052b8a0-4ac9-4ef4-b437-b62f088117e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:35:50.791957 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.791849    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8052b8a0-4ac9-4ef4-b437-b62f088117e1-kube-api-access-qbm95" (OuterVolumeSpecName: "kube-api-access-qbm95") pod "8052b8a0-4ac9-4ef4-b437-b62f088117e1" (UID: "8052b8a0-4ac9-4ef4-b437-b62f088117e1"). InnerVolumeSpecName "kube-api-access-qbm95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:35:50.890770 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.890742    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:50.890770 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.890767    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8052b8a0-4ac9-4ef4-b437-b62f088117e1-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:50.890963 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.890779    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8052b8a0-4ac9-4ef4-b437-b62f088117e1-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:50.890963 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:50.890788    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbm95\" (UniqueName: \"kubernetes.io/projected/8052b8a0-4ac9-4ef4-b437-b62f088117e1-kube-api-access-qbm95\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:35:51.419297 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.419260    2556 generic.go:356] "Generic (PLEG): container finished" podID="8052b8a0-4ac9-4ef4-b437-b62f088117e1" containerID="64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6" exitCode=0
Sep 16 02:35:51.419496 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.419352    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"
Sep 16 02:35:51.419496 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.419298    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl" event={"ID":"8052b8a0-4ac9-4ef4-b437-b62f088117e1","Type":"ContainerDied","Data":"64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6"}
Sep 16 02:35:51.419622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.419539    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl" event={"ID":"8052b8a0-4ac9-4ef4-b437-b62f088117e1","Type":"ContainerDied","Data":"deea9fe3f4c81fa788e31c195d9b8a0afaf05668b6ad843db8eb57bd6b25a0e6"}
Sep 16 02:35:51.419695 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.419679    2556 scope.go:117] "RemoveContainer" containerID="64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6"
Sep 16 02:35:51.431980 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.431954    2556 scope.go:117] "RemoveContainer" containerID="64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6"
Sep 16 02:35:51.432340 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:51.432306    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6\": container with ID starting with 64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6 not found: ID does not exist" containerID="64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6"
Sep 16 02:35:51.432340 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.432333    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6"} err="failed to get container status \"64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6\": rpc error: code = NotFound desc = could not find container \"64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6\": container with ID starting with 64e40ee8fb4e9ee53b2d2bba50ed8d219224b7119b6a41e7455b411c647069e6 not found: ID does not exist"
Sep 16 02:35:51.445357 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.445334    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"]
Sep 16 02:35:51.456213 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:51.456187    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c40df7fa-658d77f764-bphsl"]
Sep 16 02:35:53.115341 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:53.115299    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8052b8a0-4ac9-4ef4-b437-b62f088117e1" path="/var/lib/kubelet/pods/8052b8a0-4ac9-4ef4-b437-b62f088117e1/volumes"
Sep 16 02:35:55.410677 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:55.410650    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:35:56.299615 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.299571    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"]
Sep 16 02:35:56.300267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.300239    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8052b8a0-4ac9-4ef4-b437-b62f088117e1" containerName="maas-api"
Sep 16 02:35:56.300267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.300266    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8052b8a0-4ac9-4ef4-b437-b62f088117e1" containerName="maas-api"
Sep 16 02:35:56.300510 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.300364    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8052b8a0-4ac9-4ef4-b437-b62f088117e1" containerName="maas-api"
Sep 16 02:35:56.305394 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.305363    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.307744 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.307723    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b099417d-serving-cert\""
Sep 16 02:35:56.314662 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.314527    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"]
Sep 16 02:35:56.444787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.444747    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.445211 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.444793    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12cc38e-0c59-4192-8e6c-6373adbdc98e-openshift-service-ca\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.445211 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.444909    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-metrics-certs\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.445211 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.444947    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62ps\" (UniqueName: \"kubernetes.io/projected/f12cc38e-0c59-4192-8e6c-6373adbdc98e-kube-api-access-k62ps\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.546359 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.546325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.546894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.546362    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12cc38e-0c59-4192-8e6c-6373adbdc98e-openshift-service-ca\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.546894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.546420    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-metrics-certs\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.546894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.546470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62ps\" (UniqueName: \"kubernetes.io/projected/f12cc38e-0c59-4192-8e6c-6373adbdc98e-kube-api-access-k62ps\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.546894 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:56.546501    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-serving-cert: secret "maas-api-e2e-ait-b099417d-serving-cert" not found
Sep 16 02:35:56.546894 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:35:56.546634    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls podName:f12cc38e-0c59-4192-8e6c-6373adbdc98e nodeName:}" failed. No retries permitted until 2026-09-16 02:35:57.046610217 +0000 UTC m=+1056.476671593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls") pod "maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj" (UID: "f12cc38e-0c59-4192-8e6c-6373adbdc98e") : secret "maas-api-e2e-ait-b099417d-serving-cert" not found
Sep 16 02:35:56.547209 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.547142    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12cc38e-0c59-4192-8e6c-6373adbdc98e-openshift-service-ca\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.549186 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.549155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-metrics-certs\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:56.554706 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:56.554652    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62ps\" (UniqueName: \"kubernetes.io/projected/f12cc38e-0c59-4192-8e6c-6373adbdc98e-kube-api-access-k62ps\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:57.051678 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:57.051636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:57.054074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:57.054044    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls\") pod \"maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:57.219648 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:57.219610    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:57.488978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:57.488953    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"]
Sep 16 02:35:57.491388 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:35:57.491362    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12cc38e_0c59_4192_8e6c_6373adbdc98e.slice/crio-cf9bb9167def97f6ae8de44a4e9e8232c354ce45405c8c31170b618aeab7b454 WatchSource:0}: Error finding container cf9bb9167def97f6ae8de44a4e9e8232c354ce45405c8c31170b618aeab7b454: Status 404 returned error can't find the container with id cf9bb9167def97f6ae8de44a4e9e8232c354ce45405c8c31170b618aeab7b454
Sep 16 02:35:58.462466 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:58.462413    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj" event={"ID":"f12cc38e-0c59-4192-8e6c-6373adbdc98e","Type":"ContainerStarted","Data":"338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320"}
Sep 16 02:35:58.462649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:58.462574    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:35:58.462649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:58.462605    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj" event={"ID":"f12cc38e-0c59-4192-8e6c-6373adbdc98e","Type":"ContainerStarted","Data":"cf9bb9167def97f6ae8de44a4e9e8232c354ce45405c8c31170b618aeab7b454"}
Sep 16 02:35:58.477416 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:35:58.477355    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj" podStartSLOduration=2.155604529 podStartE2EDuration="2.477338282s" totalImagesPullingTime="321.733753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:35:58.476613271 +0000 UTC m=+1057.906674660" watchObservedRunningTime="2026-09-16 02:35:58.477338282 +0000 UTC m=+1057.907399705"
Sep 16 02:36:04.473636 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:04.473607    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:36:05.198332 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.198297    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"]
Sep 16 02:36:05.203549 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.203519    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.205853 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.205830    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-eee708-serving-cert\""
Sep 16 02:36:05.212307 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.212286    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"]
Sep 16 02:36:05.227709 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.227680    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.227818 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.227734    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.227818 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.227806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe9630f5-76ac-4144-9302-3626ad47d2d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.227916 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.227892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhc5\" (UniqueName: \"kubernetes.io/projected/fe9630f5-76ac-4144-9302-3626ad47d2d4-kube-api-access-fbhc5\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.329189 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.329154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe9630f5-76ac-4144-9302-3626ad47d2d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.329367 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.329209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhc5\" (UniqueName: \"kubernetes.io/projected/fe9630f5-76ac-4144-9302-3626ad47d2d4-kube-api-access-fbhc5\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.329367 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.329264    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.329367 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.329296    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.330000 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.329966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe9630f5-76ac-4144-9302-3626ad47d2d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.331803 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.331771    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.331926 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.331874    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.337858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.337835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhc5\" (UniqueName: \"kubernetes.io/projected/fe9630f5-76ac-4144-9302-3626ad47d2d4-kube-api-access-fbhc5\") pod \"maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.517832 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.517745    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:05.800120 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:05.800092    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"]
Sep 16 02:36:05.801947 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:36:05.801910    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9630f5_76ac_4144_9302_3626ad47d2d4.slice/crio-9685310ad17671923b15cd2d86035895dcb86c8a198e595dbaec722b303dd50d WatchSource:0}: Error finding container 9685310ad17671923b15cd2d86035895dcb86c8a198e595dbaec722b303dd50d: Status 404 returned error can't find the container with id 9685310ad17671923b15cd2d86035895dcb86c8a198e595dbaec722b303dd50d
Sep 16 02:36:06.509387 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:06.509242    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd" event={"ID":"fe9630f5-76ac-4144-9302-3626ad47d2d4","Type":"ContainerStarted","Data":"3d7d83abc6435ccf6cd422cb2dd1bdde5ce2922c5ea243816251b36821173fdc"}
Sep 16 02:36:06.509387 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:06.509317    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd" event={"ID":"fe9630f5-76ac-4144-9302-3626ad47d2d4","Type":"ContainerStarted","Data":"9685310ad17671923b15cd2d86035895dcb86c8a198e595dbaec722b303dd50d"}
Sep 16 02:36:06.509631 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:06.509397    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:12.519299 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:12.519269    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:36:12.535562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:12.535517    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd" podStartSLOduration=7.137627415 podStartE2EDuration="7.535500508s" totalImagesPullingTime="397.873093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:06.532696072 +0000 UTC m=+1065.962757458" watchObservedRunningTime="2026-09-16 02:36:12.535500508 +0000 UTC m=+1071.965561892"
Sep 16 02:36:26.189908 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.189868    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"]
Sep 16 02:36:26.190470 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.190132    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj" podUID="f12cc38e-0c59-4192-8e6c-6373adbdc98e" containerName="maas-api" containerID="cri-o://338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320" gracePeriod=30
Sep 16 02:36:26.448632 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.448570    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:36:26.524973 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.524937    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls\") pod \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") "
Sep 16 02:36:26.525139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.524989    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k62ps\" (UniqueName: \"kubernetes.io/projected/f12cc38e-0c59-4192-8e6c-6373adbdc98e-kube-api-access-k62ps\") pod \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") "
Sep 16 02:36:26.525139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.525038    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12cc38e-0c59-4192-8e6c-6373adbdc98e-openshift-service-ca\") pod \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") "
Sep 16 02:36:26.525139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.525100    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-metrics-certs\") pod \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\" (UID: \"f12cc38e-0c59-4192-8e6c-6373adbdc98e\") "
Sep 16 02:36:26.525503 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.525471    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12cc38e-0c59-4192-8e6c-6373adbdc98e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f12cc38e-0c59-4192-8e6c-6373adbdc98e" (UID: "f12cc38e-0c59-4192-8e6c-6373adbdc98e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:36:26.527185 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.527134    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f12cc38e-0c59-4192-8e6c-6373adbdc98e" (UID: "f12cc38e-0c59-4192-8e6c-6373adbdc98e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:36:26.527185 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.527148    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f12cc38e-0c59-4192-8e6c-6373adbdc98e" (UID: "f12cc38e-0c59-4192-8e6c-6373adbdc98e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:36:26.527185 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.527165    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12cc38e-0c59-4192-8e6c-6373adbdc98e-kube-api-access-k62ps" (OuterVolumeSpecName: "kube-api-access-k62ps") pod "f12cc38e-0c59-4192-8e6c-6373adbdc98e" (UID: "f12cc38e-0c59-4192-8e6c-6373adbdc98e"). InnerVolumeSpecName "kube-api-access-k62ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:36:26.621148 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.621114    2556 generic.go:356] "Generic (PLEG): container finished" podID="f12cc38e-0c59-4192-8e6c-6373adbdc98e" containerID="338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320" exitCode=0
Sep 16 02:36:26.621285 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.621168    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"
Sep 16 02:36:26.621285 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.621196    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj" event={"ID":"f12cc38e-0c59-4192-8e6c-6373adbdc98e","Type":"ContainerDied","Data":"338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320"}
Sep 16 02:36:26.621285 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.621257    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj" event={"ID":"f12cc38e-0c59-4192-8e6c-6373adbdc98e","Type":"ContainerDied","Data":"cf9bb9167def97f6ae8de44a4e9e8232c354ce45405c8c31170b618aeab7b454"}
Sep 16 02:36:26.621405 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.621288    2556 scope.go:117] "RemoveContainer" containerID="338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320"
Sep 16 02:36:26.625885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.625865    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.625885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.625885    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f12cc38e-0c59-4192-8e6c-6373adbdc98e-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.626040 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.625893    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62ps\" (UniqueName: \"kubernetes.io/projected/f12cc38e-0c59-4192-8e6c-6373adbdc98e-kube-api-access-k62ps\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.626040 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.625903    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f12cc38e-0c59-4192-8e6c-6373adbdc98e-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:36:26.631028 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.631008    2556 scope.go:117] "RemoveContainer" containerID="338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320"
Sep 16 02:36:26.631302 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:36:26.631282    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320\": container with ID starting with 338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320 not found: ID does not exist" containerID="338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320"
Sep 16 02:36:26.631350 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.631312    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320"} err="failed to get container status \"338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320\": rpc error: code = NotFound desc = could not find container \"338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320\": container with ID starting with 338792831c355196d2cff819d4d1a0e526e7f1b93bdfab0721688b12ebbd3320 not found: ID does not exist"
Sep 16 02:36:26.642059 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.642037    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"]
Sep 16 02:36:26.645608 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:26.645588    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b099417d-554f65fbc4-x4rpj"]
Sep 16 02:36:27.114024 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:27.113989    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12cc38e-0c59-4192-8e6c-6373adbdc98e" path="/var/lib/kubelet/pods/f12cc38e-0c59-4192-8e6c-6373adbdc98e/volumes"
Sep 16 02:36:33.633620 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.633579    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"]
Sep 16 02:36:33.634268 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.634249    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12cc38e-0c59-4192-8e6c-6373adbdc98e" containerName="maas-api"
Sep 16 02:36:33.634342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.634271    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12cc38e-0c59-4192-8e6c-6373adbdc98e" containerName="maas-api"
Sep 16 02:36:33.634398 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.634386    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12cc38e-0c59-4192-8e6c-6373adbdc98e" containerName="maas-api"
Sep 16 02:36:33.638287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.638263    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.640393 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.640367    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-642a8155-serving-cert\""
Sep 16 02:36:33.646797 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.646767    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"]
Sep 16 02:36:33.799963 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.799919    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-maas-api-tls\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.800162 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.799982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015a8baa-9aad-4484-b6a0-b266b3842e50-openshift-service-ca\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.800162 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.800012    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-metrics-certs\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.800162 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.800125    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhc9\" (UniqueName: \"kubernetes.io/projected/015a8baa-9aad-4484-b6a0-b266b3842e50-kube-api-access-pfhc9\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.901399 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.901302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-maas-api-tls\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.901399 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.901355    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015a8baa-9aad-4484-b6a0-b266b3842e50-openshift-service-ca\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.901399 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.901386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-metrics-certs\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.901696 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.901464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhc9\" (UniqueName: \"kubernetes.io/projected/015a8baa-9aad-4484-b6a0-b266b3842e50-kube-api-access-pfhc9\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.902222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.902192    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015a8baa-9aad-4484-b6a0-b266b3842e50-openshift-service-ca\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.904402 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.904373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-metrics-certs\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.904574 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.904551    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-maas-api-tls\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.909822 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.909795    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhc9\" (UniqueName: \"kubernetes.io/projected/015a8baa-9aad-4484-b6a0-b266b3842e50-kube-api-access-pfhc9\") pod \"maas-api-e2e-derive-642a8155-6ff5484979-g5hpz\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:33.952666 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:33.952630    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:34.237706 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:34.237669    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"]
Sep 16 02:36:34.239781 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:36:34.239741    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015a8baa_9aad_4484_b6a0_b266b3842e50.slice/crio-5b23490b26f5b3845a2c08eb64ef4e32f643cd34fd229b0b8d09fa16ca76d4a1 WatchSource:0}: Error finding container 5b23490b26f5b3845a2c08eb64ef4e32f643cd34fd229b0b8d09fa16ca76d4a1: Status 404 returned error can't find the container with id 5b23490b26f5b3845a2c08eb64ef4e32f643cd34fd229b0b8d09fa16ca76d4a1
Sep 16 02:36:34.668031 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:34.668002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz" event={"ID":"015a8baa-9aad-4484-b6a0-b266b3842e50","Type":"ContainerStarted","Data":"5b23490b26f5b3845a2c08eb64ef4e32f643cd34fd229b0b8d09fa16ca76d4a1"}
Sep 16 02:36:35.675503 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:35.675457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz" event={"ID":"015a8baa-9aad-4484-b6a0-b266b3842e50","Type":"ContainerStarted","Data":"716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b"}
Sep 16 02:36:35.675903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:35.675534    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:36:35.692759 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:35.692709    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz" podStartSLOduration=2.199092259 podStartE2EDuration="2.69269411s" totalImagesPullingTime="493.601851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:36:35.690035228 +0000 UTC m=+1095.120096613" watchObservedRunningTime="2026-09-16 02:36:35.69269411 +0000 UTC m=+1095.122755544"
Sep 16 02:36:40.107481 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:40.107449    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:36:41.687903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:36:41.687872    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:37:08.967423 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:08.967379    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"]
Sep 16 02:37:09.555990 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.555957    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"]
Sep 16 02:37:09.559970 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.559948    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.563397 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.563369    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"kube-root-ca.crt\""
Sep 16 02:37:09.563567 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.563554    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"openshift-service-ca.crt\""
Sep 16 02:37:09.563639 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.563582    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"default-dockercfg-2w275\""
Sep 16 02:37:09.563639 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.563598    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"auth-test-model-eee708-kserve-self-signed-certs\""
Sep 16 02:37:09.568277 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.568252    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"]
Sep 16 02:37:09.656444 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.656397    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tls-certs\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.656661 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.656509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-model-cache\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.656661 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.656549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-dshm\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.656661 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.656607    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxrf\" (UniqueName: \"kubernetes.io/projected/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-kube-api-access-pfxrf\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.656834 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.656721    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tmp-dir\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.656834 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.656805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-home\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.757850 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.757803    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tmp-dir\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758058 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.757884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-home\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758058 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.757948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tls-certs\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758058 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.757979    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-model-cache\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758058 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.758006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-dshm\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758058 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.758051    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxrf\" (UniqueName: \"kubernetes.io/projected/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-kube-api-access-pfxrf\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758296 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.758271    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tmp-dir\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.758306    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-home\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.758417 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.758384    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-model-cache\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.760301 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.760276    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-dshm\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.760460 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.760441    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tls-certs\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.765608 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.765589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxrf\" (UniqueName: \"kubernetes.io/projected/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-kube-api-access-pfxrf\") pod \"auth-test-model-eee708-kserve-65478b77c9-5phl4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") " pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:09.872577 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:09.872486    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:10.144167 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:10.144141    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"]
Sep 16 02:37:10.146415 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:37:10.146387    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64ed0f31_36ec_4c93_aa00_318e4b2f69a4.slice/crio-765f9f2815a7d3dfcd01c30c977bee5d90bd64a0e7709a97b067bb9ce324beb3 WatchSource:0}: Error finding container 765f9f2815a7d3dfcd01c30c977bee5d90bd64a0e7709a97b067bb9ce324beb3: Status 404 returned error can't find the container with id 765f9f2815a7d3dfcd01c30c977bee5d90bd64a0e7709a97b067bb9ce324beb3
Sep 16 02:37:10.863895 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:10.863858    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4" event={"ID":"64ed0f31-36ec-4c93-aa00-318e4b2f69a4","Type":"ContainerStarted","Data":"e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e"}
Sep 16 02:37:10.863895 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:10.863901    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4" event={"ID":"64ed0f31-36ec-4c93-aa00-318e4b2f69a4","Type":"ContainerStarted","Data":"765f9f2815a7d3dfcd01c30c977bee5d90bd64a0e7709a97b067bb9ce324beb3"}
Sep 16 02:37:10.880568 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:10.880510    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4" podStartSLOduration=1.880493035 podStartE2EDuration="1.880493035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:37:10.878393928 +0000 UTC m=+1130.308455314" watchObservedRunningTime="2026-09-16 02:37:10.880493035 +0000 UTC m=+1130.310554421"
Sep 16 02:37:12.326297 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.326265    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"]
Sep 16 02:37:12.326802 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.326556    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz" podUID="015a8baa-9aad-4484-b6a0-b266b3842e50" containerName="maas-api" containerID="cri-o://716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b" gracePeriod=30
Sep 16 02:37:12.584752 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.584685    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:37:12.686013 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.685984    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015a8baa-9aad-4484-b6a0-b266b3842e50-openshift-service-ca\") pod \"015a8baa-9aad-4484-b6a0-b266b3842e50\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") "
Sep 16 02:37:12.686013 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.686018    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfhc9\" (UniqueName: \"kubernetes.io/projected/015a8baa-9aad-4484-b6a0-b266b3842e50-kube-api-access-pfhc9\") pod \"015a8baa-9aad-4484-b6a0-b266b3842e50\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") "
Sep 16 02:37:12.686283 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.686049    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-metrics-certs\") pod \"015a8baa-9aad-4484-b6a0-b266b3842e50\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") "
Sep 16 02:37:12.686283 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.686129    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-maas-api-tls\") pod \"015a8baa-9aad-4484-b6a0-b266b3842e50\" (UID: \"015a8baa-9aad-4484-b6a0-b266b3842e50\") "
Sep 16 02:37:12.686405 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.686365    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015a8baa-9aad-4484-b6a0-b266b3842e50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "015a8baa-9aad-4484-b6a0-b266b3842e50" (UID: "015a8baa-9aad-4484-b6a0-b266b3842e50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:12.686544 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.686529    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/015a8baa-9aad-4484-b6a0-b266b3842e50-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:12.688116 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.688089    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "015a8baa-9aad-4484-b6a0-b266b3842e50" (UID: "015a8baa-9aad-4484-b6a0-b266b3842e50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:12.688216 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.688131    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "015a8baa-9aad-4484-b6a0-b266b3842e50" (UID: "015a8baa-9aad-4484-b6a0-b266b3842e50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:12.688216 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.688172    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015a8baa-9aad-4484-b6a0-b266b3842e50-kube-api-access-pfhc9" (OuterVolumeSpecName: "kube-api-access-pfhc9") pod "015a8baa-9aad-4484-b6a0-b266b3842e50" (UID: "015a8baa-9aad-4484-b6a0-b266b3842e50"). InnerVolumeSpecName "kube-api-access-pfhc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:12.787928 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.787891    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfhc9\" (UniqueName: \"kubernetes.io/projected/015a8baa-9aad-4484-b6a0-b266b3842e50-kube-api-access-pfhc9\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:12.787928 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.787922    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:12.787928 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.787932    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/015a8baa-9aad-4484-b6a0-b266b3842e50-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:12.875066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.874967    2556 generic.go:356] "Generic (PLEG): container finished" podID="015a8baa-9aad-4484-b6a0-b266b3842e50" containerID="716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b" exitCode=0
Sep 16 02:37:12.875066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.875012    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"
Sep 16 02:37:12.875066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.875037    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz" event={"ID":"015a8baa-9aad-4484-b6a0-b266b3842e50","Type":"ContainerDied","Data":"716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b"}
Sep 16 02:37:12.875337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.875080    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz" event={"ID":"015a8baa-9aad-4484-b6a0-b266b3842e50","Type":"ContainerDied","Data":"5b23490b26f5b3845a2c08eb64ef4e32f643cd34fd229b0b8d09fa16ca76d4a1"}
Sep 16 02:37:12.875337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.875102    2556 scope.go:117] "RemoveContainer" containerID="716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b"
Sep 16 02:37:12.885088 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.885068    2556 scope.go:117] "RemoveContainer" containerID="716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b"
Sep 16 02:37:12.885386 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:37:12.885365    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b\": container with ID starting with 716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b not found: ID does not exist" containerID="716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b"
Sep 16 02:37:12.885457 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.885396    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b"} err="failed to get container status \"716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b\": rpc error: code = NotFound desc = could not find container \"716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b\": container with ID starting with 716bdc418d75cdc3de6d2b24c164b2eb5e3fb6297e8dbda30862892cca68e61b not found: ID does not exist"
Sep 16 02:37:12.895676 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.895652    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"]
Sep 16 02:37:12.902387 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:12.902362    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-642a8155-6ff5484979-g5hpz"]
Sep 16 02:37:13.113899 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:13.113864    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="015a8baa-9aad-4484-b6a0-b266b3842e50" path="/var/lib/kubelet/pods/015a8baa-9aad-4484-b6a0-b266b3842e50/volumes"
Sep 16 02:37:19.872714 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:19.872679    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:19.872714 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:19.872723    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:19.885524 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:19.885494    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:19.924504 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:19.924475    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:37:21.151851 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.151817    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"]
Sep 16 02:37:21.152349 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.152113    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q" podUID="90f8e17e-7c2a-4415-b101-a8357bd0ac33" containerName="maas-api" containerID="cri-o://fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850" gracePeriod=30
Sep 16 02:37:21.615486 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.615412    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:37:21.779676 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.779580    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls\") pod \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") "
Sep 16 02:37:21.779676 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.779636    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mklx\" (UniqueName: \"kubernetes.io/projected/90f8e17e-7c2a-4415-b101-a8357bd0ac33-kube-api-access-2mklx\") pod \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") "
Sep 16 02:37:21.779854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.779725    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f8e17e-7c2a-4415-b101-a8357bd0ac33-openshift-service-ca\") pod \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") "
Sep 16 02:37:21.779854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.779783    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-metrics-certs\") pod \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\" (UID: \"90f8e17e-7c2a-4415-b101-a8357bd0ac33\") "
Sep 16 02:37:21.780164 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.780136    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f8e17e-7c2a-4415-b101-a8357bd0ac33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90f8e17e-7c2a-4415-b101-a8357bd0ac33" (UID: "90f8e17e-7c2a-4415-b101-a8357bd0ac33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:37:21.781866 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.781839    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90f8e17e-7c2a-4415-b101-a8357bd0ac33" (UID: "90f8e17e-7c2a-4415-b101-a8357bd0ac33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:21.781966 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.781900    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90f8e17e-7c2a-4415-b101-a8357bd0ac33" (UID: "90f8e17e-7c2a-4415-b101-a8357bd0ac33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:21.781966 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.781915    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f8e17e-7c2a-4415-b101-a8357bd0ac33-kube-api-access-2mklx" (OuterVolumeSpecName: "kube-api-access-2mklx") pod "90f8e17e-7c2a-4415-b101-a8357bd0ac33" (UID: "90f8e17e-7c2a-4415-b101-a8357bd0ac33"). InnerVolumeSpecName "kube-api-access-2mklx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:21.881348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.881316    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f8e17e-7c2a-4415-b101-a8357bd0ac33-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:21.881348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.881346    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:21.881348 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.881354    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90f8e17e-7c2a-4415-b101-a8357bd0ac33-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:21.881611 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.881362    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mklx\" (UniqueName: \"kubernetes.io/projected/90f8e17e-7c2a-4415-b101-a8357bd0ac33-kube-api-access-2mklx\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:21.925371 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.925336    2556 generic.go:356] "Generic (PLEG): container finished" podID="90f8e17e-7c2a-4415-b101-a8357bd0ac33" containerID="fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850" exitCode=0
Sep 16 02:37:21.925561 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.925381    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"
Sep 16 02:37:21.925561 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.925414    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q" event={"ID":"90f8e17e-7c2a-4415-b101-a8357bd0ac33","Type":"ContainerDied","Data":"fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850"}
Sep 16 02:37:21.925561 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.925484    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q" event={"ID":"90f8e17e-7c2a-4415-b101-a8357bd0ac33","Type":"ContainerDied","Data":"ef904cdf1fe13a8da329bd99aa41ec50c218f3a7cc84e3592beca9e87d569f8a"}
Sep 16 02:37:21.925561 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.925509    2556 scope.go:117] "RemoveContainer" containerID="fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850"
Sep 16 02:37:21.935521 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.935500    2556 scope.go:117] "RemoveContainer" containerID="fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850"
Sep 16 02:37:21.935812 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:37:21.935791    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850\": container with ID starting with fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850 not found: ID does not exist" containerID="fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850"
Sep 16 02:37:21.935863 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.935824    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850"} err="failed to get container status \"fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850\": rpc error: code = NotFound desc = could not find container \"fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850\": container with ID starting with fc755c3cc8f2f5ab291d3c6c20478e34c45472bd52dba2fca903aa6ab1fbe850 not found: ID does not exist"
Sep 16 02:37:21.946766 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.946737    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"]
Sep 16 02:37:21.951095 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:21.951050    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a3d7eb-7ff95864b5-md68q"]
Sep 16 02:37:23.116060 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:23.116022    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f8e17e-7c2a-4415-b101-a8357bd0ac33" path="/var/lib/kubelet/pods/90f8e17e-7c2a-4415-b101-a8357bd0ac33/volumes"
Sep 16 02:37:23.987223 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:23.987183    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz" podUID="9514d97e-2329-4757-aa9c-eb06ca82b309" containerName="main" containerID="cri-o://e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d" gracePeriod=45
Sep 16 02:37:24.245771 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.245708    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:37:24.319720 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.319686    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-dshm\") pod \"9514d97e-2329-4757-aa9c-eb06ca82b309\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") "
Sep 16 02:37:24.319902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.319775    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj4tn\" (UniqueName: \"kubernetes.io/projected/9514d97e-2329-4757-aa9c-eb06ca82b309-kube-api-access-jj4tn\") pod \"9514d97e-2329-4757-aa9c-eb06ca82b309\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") "
Sep 16 02:37:24.319902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.319806    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-tmp-dir\") pod \"9514d97e-2329-4757-aa9c-eb06ca82b309\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") "
Sep 16 02:37:24.319902 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.319850    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-model-cache\") pod \"9514d97e-2329-4757-aa9c-eb06ca82b309\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") "
Sep 16 02:37:24.320070 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.319937    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9514d97e-2329-4757-aa9c-eb06ca82b309-tls-certs\") pod \"9514d97e-2329-4757-aa9c-eb06ca82b309\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") "
Sep 16 02:37:24.320070 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.319963    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-home\") pod \"9514d97e-2329-4757-aa9c-eb06ca82b309\" (UID: \"9514d97e-2329-4757-aa9c-eb06ca82b309\") "
Sep 16 02:37:24.320501 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.320415    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-model-cache" (OuterVolumeSpecName: "model-cache") pod "9514d97e-2329-4757-aa9c-eb06ca82b309" (UID: "9514d97e-2329-4757-aa9c-eb06ca82b309"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.320601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.320515    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-home" (OuterVolumeSpecName: "home") pod "9514d97e-2329-4757-aa9c-eb06ca82b309" (UID: "9514d97e-2329-4757-aa9c-eb06ca82b309"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.320601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.320572    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9514d97e-2329-4757-aa9c-eb06ca82b309" (UID: "9514d97e-2329-4757-aa9c-eb06ca82b309"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.322313 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.322281    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9514d97e-2329-4757-aa9c-eb06ca82b309-kube-api-access-jj4tn" (OuterVolumeSpecName: "kube-api-access-jj4tn") pod "9514d97e-2329-4757-aa9c-eb06ca82b309" (UID: "9514d97e-2329-4757-aa9c-eb06ca82b309"). InnerVolumeSpecName "kube-api-access-jj4tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:37:24.322410 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.322330    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-dshm" (OuterVolumeSpecName: "dshm") pod "9514d97e-2329-4757-aa9c-eb06ca82b309" (UID: "9514d97e-2329-4757-aa9c-eb06ca82b309"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:37:24.322573 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.322507    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9514d97e-2329-4757-aa9c-eb06ca82b309-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9514d97e-2329-4757-aa9c-eb06ca82b309" (UID: "9514d97e-2329-4757-aa9c-eb06ca82b309"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:37:24.421560 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.421529    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.421560 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.421556    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9514d97e-2329-4757-aa9c-eb06ca82b309-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.421765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.421569    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.421765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.421579    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.421765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.421592    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj4tn\" (UniqueName: \"kubernetes.io/projected/9514d97e-2329-4757-aa9c-eb06ca82b309-kube-api-access-jj4tn\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.421765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.421603    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9514d97e-2329-4757-aa9c-eb06ca82b309-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:37:24.944001 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.943963    2556 generic.go:356] "Generic (PLEG): container finished" podID="9514d97e-2329-4757-aa9c-eb06ca82b309" containerID="e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d" exitCode=0
Sep 16 02:37:24.944222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.944009    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"
Sep 16 02:37:24.944222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.944051    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz" event={"ID":"9514d97e-2329-4757-aa9c-eb06ca82b309","Type":"ContainerDied","Data":"e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d"}
Sep 16 02:37:24.944222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.944098    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz" event={"ID":"9514d97e-2329-4757-aa9c-eb06ca82b309","Type":"ContainerDied","Data":"42fc317ec9264f820408c6bf3c5f458a52ec8cdb8d27fb6f176c921177633657"}
Sep 16 02:37:24.944222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.944116    2556 scope.go:117] "RemoveContainer" containerID="e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d"
Sep 16 02:37:24.954120 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.954099    2556 scope.go:117] "RemoveContainer" containerID="e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d"
Sep 16 02:37:24.954460 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:37:24.954423    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d\": container with ID starting with e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d not found: ID does not exist" containerID="e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d"
Sep 16 02:37:24.954549 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.954473    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d"} err="failed to get container status \"e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d\": rpc error: code = NotFound desc = could not find container \"e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d\": container with ID starting with e67c3f5ffad901baea594c2ad9e1e45e85ba5d69de5e108eb1fef49fc33fe42d not found: ID does not exist"
Sep 16 02:37:24.965205 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.965177    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"]
Sep 16 02:37:24.970841 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:24.970812    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a3d7eb/premium-w3-a3d7eb-kserve-7cb56946c5-7nnbz"]
Sep 16 02:37:25.113954 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:37:25.113919    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9514d97e-2329-4757-aa9c-eb06ca82b309" path="/var/lib/kubelet/pods/9514d97e-2329-4757-aa9c-eb06ca82b309/volumes"
Sep 16 02:38:00.108256 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:00.108223    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:38:21.029453 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:21.029402    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:38:21.033178 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:21.033158    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:38:34.026404 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:34.026370    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"]
Sep 16 02:38:34.061133 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:34.061103    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"]
Sep 16 02:38:44.492276 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:44.492239    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"]
Sep 16 02:38:44.492801 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:44.492508    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms" podUID="5642889a-1592-4949-9fcd-215e3656111b" containerName="maas-api" containerID="cri-o://05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f" gracePeriod=30
Sep 16 02:38:44.952346 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:44.952317    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:38:45.073889 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.073796    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5642889a-1592-4949-9fcd-215e3656111b-openshift-service-ca\") pod \"5642889a-1592-4949-9fcd-215e3656111b\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") "
Sep 16 02:38:45.073889 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.073868    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-metrics-certs\") pod \"5642889a-1592-4949-9fcd-215e3656111b\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") "
Sep 16 02:38:45.074131 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.073909    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls\") pod \"5642889a-1592-4949-9fcd-215e3656111b\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") "
Sep 16 02:38:45.074131 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.073961    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtjck\" (UniqueName: \"kubernetes.io/projected/5642889a-1592-4949-9fcd-215e3656111b-kube-api-access-qtjck\") pod \"5642889a-1592-4949-9fcd-215e3656111b\" (UID: \"5642889a-1592-4949-9fcd-215e3656111b\") "
Sep 16 02:38:45.074232 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.074203    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5642889a-1592-4949-9fcd-215e3656111b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5642889a-1592-4949-9fcd-215e3656111b" (UID: "5642889a-1592-4949-9fcd-215e3656111b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:38:45.076048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.076019    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5642889a-1592-4949-9fcd-215e3656111b" (UID: "5642889a-1592-4949-9fcd-215e3656111b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:38:45.076180 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.076062    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5642889a-1592-4949-9fcd-215e3656111b" (UID: "5642889a-1592-4949-9fcd-215e3656111b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:38:45.076180 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.076089    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5642889a-1592-4949-9fcd-215e3656111b-kube-api-access-qtjck" (OuterVolumeSpecName: "kube-api-access-qtjck") pod "5642889a-1592-4949-9fcd-215e3656111b" (UID: "5642889a-1592-4949-9fcd-215e3656111b"). InnerVolumeSpecName "kube-api-access-qtjck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:38:45.175665 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.175629    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:45.175665 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.175658    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5642889a-1592-4949-9fcd-215e3656111b-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:45.175665 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.175667    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtjck\" (UniqueName: \"kubernetes.io/projected/5642889a-1592-4949-9fcd-215e3656111b-kube-api-access-qtjck\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:45.175901 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.175677    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5642889a-1592-4949-9fcd-215e3656111b-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:45.365108 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.365013    2556 generic.go:356] "Generic (PLEG): container finished" podID="5642889a-1592-4949-9fcd-215e3656111b" containerID="05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f" exitCode=0
Sep 16 02:38:45.365108 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.365065    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"
Sep 16 02:38:45.365331 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.365098    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms" event={"ID":"5642889a-1592-4949-9fcd-215e3656111b","Type":"ContainerDied","Data":"05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f"}
Sep 16 02:38:45.365331 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.365191    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms" event={"ID":"5642889a-1592-4949-9fcd-215e3656111b","Type":"ContainerDied","Data":"c7f69e888ab984a7f24cf4b05188deb306c83434a3f40d65f2016f3200b14a44"}
Sep 16 02:38:45.365331 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.365227    2556 scope.go:117] "RemoveContainer" containerID="05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f"
Sep 16 02:38:45.375555 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.375535    2556 scope.go:117] "RemoveContainer" containerID="05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f"
Sep 16 02:38:45.375843 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:38:45.375820    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f\": container with ID starting with 05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f not found: ID does not exist" containerID="05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f"
Sep 16 02:38:45.375904 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.375851    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f"} err="failed to get container status \"05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f\": rpc error: code = NotFound desc = could not find container \"05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f\": container with ID starting with 05732968c5d1ce711219b27c60d75342a1b6076986ae6cde2f6361212bdfd46f not found: ID does not exist"
Sep 16 02:38:45.383584 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.383560    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"]
Sep 16 02:38:45.389291 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:45.389268    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5b59aa-5cbdd7664c-dm8ms"]
Sep 16 02:38:47.115572 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:47.115534    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5642889a-1592-4949-9fcd-215e3656111b" path="/var/lib/kubelet/pods/5642889a-1592-4949-9fcd-215e3656111b/volumes"
Sep 16 02:38:49.046666 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.046631    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4" podUID="1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" containerName="main" containerID="cri-o://6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2" gracePeriod=45
Sep 16 02:38:49.080963 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.080931    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb" podUID="55adf607-21aa-40ef-b1e6-8e8ba8232c85" containerName="main" containerID="cri-o://67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f" gracePeriod=45
Sep 16 02:38:49.359543 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.359518    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:38:49.362590 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.362569    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:38:49.392968 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.392934    2556 generic.go:356] "Generic (PLEG): container finished" podID="55adf607-21aa-40ef-b1e6-8e8ba8232c85" containerID="67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f" exitCode=0
Sep 16 02:38:49.393313 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.392974    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"
Sep 16 02:38:49.393313 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.392990    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb" event={"ID":"55adf607-21aa-40ef-b1e6-8e8ba8232c85","Type":"ContainerDied","Data":"67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f"}
Sep 16 02:38:49.393619 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.393350    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb" event={"ID":"55adf607-21aa-40ef-b1e6-8e8ba8232c85","Type":"ContainerDied","Data":"89d0dd568111d6ac0249cadfb141e0dc4df7179d55586852167a37c55a2816ba"}
Sep 16 02:38:49.393619 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.393466    2556 scope.go:117] "RemoveContainer" containerID="67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f"
Sep 16 02:38:49.395309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.395230    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" containerID="6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2" exitCode=0
Sep 16 02:38:49.395309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.395288    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"
Sep 16 02:38:49.395470 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.395282    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4" event={"ID":"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb","Type":"ContainerDied","Data":"6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2"}
Sep 16 02:38:49.395539 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.395483    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4" event={"ID":"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb","Type":"ContainerDied","Data":"7b0e1324828cf850c162a194df1b3bb57d7d682d52e17346585c52a6a396fc4f"}
Sep 16 02:38:49.406419 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.406402    2556 scope.go:117] "RemoveContainer" containerID="67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f"
Sep 16 02:38:49.406787 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:38:49.406767    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f\": container with ID starting with 67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f not found: ID does not exist" containerID="67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f"
Sep 16 02:38:49.406885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.406793    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f"} err="failed to get container status \"67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f\": rpc error: code = NotFound desc = could not find container \"67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f\": container with ID starting with 67394c5542adc3932914e36159cb2c92de603731c0eafccf22da3980f0e7ea9f not found: ID does not exist"
Sep 16 02:38:49.406885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.406809    2556 scope.go:117] "RemoveContainer" containerID="6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2"
Sep 16 02:38:49.416597 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.416577    2556 scope.go:117] "RemoveContainer" containerID="6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2"
Sep 16 02:38:49.416919 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:38:49.416893    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2\": container with ID starting with 6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2 not found: ID does not exist" containerID="6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2"
Sep 16 02:38:49.417021 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.416927    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2"} err="failed to get container status \"6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2\": rpc error: code = NotFound desc = could not find container \"6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2\": container with ID starting with 6af77791b0a68c7bda75f8e3ba7882d65b7eb677ddcf1c45efb4d6b5a92cbae2 not found: ID does not exist"
Sep 16 02:38:49.521688 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521652    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-dshm\") pod \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") "
Sep 16 02:38:49.521688 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521693    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qvsp\" (UniqueName: \"kubernetes.io/projected/55adf607-21aa-40ef-b1e6-8e8ba8232c85-kube-api-access-2qvsp\") pod \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521716    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tmp-dir\") pod \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521732    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-home\") pod \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521757    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tls-certs\") pod \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521776    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-model-cache\") pod \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521800    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tls-certs\") pod \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521824    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tmp-dir\") pod \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521860    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-model-cache\") pod \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") "
Sep 16 02:38:49.521930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521897    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfnkx\" (UniqueName: \"kubernetes.io/projected/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-kube-api-access-qfnkx\") pod \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") "
Sep 16 02:38:49.522342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521937    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-home\") pod \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\" (UID: \"1ebcd016-962e-45c6-a0a9-5d1d909cb1cb\") "
Sep 16 02:38:49.522342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521995    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-dshm\") pod \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\" (UID: \"55adf607-21aa-40ef-b1e6-8e8ba8232c85\") "
Sep 16 02:38:49.522342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.521980    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55adf607-21aa-40ef-b1e6-8e8ba8232c85" (UID: "55adf607-21aa-40ef-b1e6-8e8ba8232c85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.522342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522063    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" (UID: "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.522342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522231    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-home" (OuterVolumeSpecName: "home") pod "55adf607-21aa-40ef-b1e6-8e8ba8232c85" (UID: "55adf607-21aa-40ef-b1e6-8e8ba8232c85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.522342 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522265    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-model-cache" (OuterVolumeSpecName: "model-cache") pod "55adf607-21aa-40ef-b1e6-8e8ba8232c85" (UID: "55adf607-21aa-40ef-b1e6-8e8ba8232c85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.522621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522391    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.522621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522410    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.522621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522423    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.522621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522470    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.522621 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522472    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" (UID: "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.522847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.522670    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-home" (OuterVolumeSpecName: "home") pod "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" (UID: "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.524222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.524195    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55adf607-21aa-40ef-b1e6-8e8ba8232c85" (UID: "55adf607-21aa-40ef-b1e6-8e8ba8232c85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:38:49.524480 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.524457    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-dshm" (OuterVolumeSpecName: "dshm") pod "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" (UID: "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.524580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.524482    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" (UID: "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:38:49.524580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.524508    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55adf607-21aa-40ef-b1e6-8e8ba8232c85-kube-api-access-2qvsp" (OuterVolumeSpecName: "kube-api-access-2qvsp") pod "55adf607-21aa-40ef-b1e6-8e8ba8232c85" (UID: "55adf607-21aa-40ef-b1e6-8e8ba8232c85"). InnerVolumeSpecName "kube-api-access-2qvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:38:49.524580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.524519    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-kube-api-access-qfnkx" (OuterVolumeSpecName: "kube-api-access-qfnkx") pod "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" (UID: "1ebcd016-962e-45c6-a0a9-5d1d909cb1cb"). InnerVolumeSpecName "kube-api-access-qfnkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:38:49.524730 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.524716    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-dshm" (OuterVolumeSpecName: "dshm") pod "55adf607-21aa-40ef-b1e6-8e8ba8232c85" (UID: "55adf607-21aa-40ef-b1e6-8e8ba8232c85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623565    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55adf607-21aa-40ef-b1e6-8e8ba8232c85-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623595    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623606    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfnkx\" (UniqueName: \"kubernetes.io/projected/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-kube-api-access-qfnkx\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623615    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623624    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55adf607-21aa-40ef-b1e6-8e8ba8232c85-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623633    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623640    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qvsp\" (UniqueName: \"kubernetes.io/projected/55adf607-21aa-40ef-b1e6-8e8ba8232c85-kube-api-access-2qvsp\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.623649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.623649    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:38:49.716132 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.716097    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"]
Sep 16 02:38:49.720018 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.719988    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/simulator-w0-5b59aa-kserve-5f984f98cb-mrmwb"]
Sep 16 02:38:49.728920 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.728888    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"]
Sep 16 02:38:49.733227 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:49.733196    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5b59aa/distinct-2-w0-5b59aa-kserve-59696d76d7-2wvv4"]
Sep 16 02:38:51.113876 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.113843    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" path="/var/lib/kubelet/pods/1ebcd016-962e-45c6-a0a9-5d1d909cb1cb/volumes"
Sep 16 02:38:51.114304 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.114290    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55adf607-21aa-40ef-b1e6-8e8ba8232c85" path="/var/lib/kubelet/pods/55adf607-21aa-40ef-b1e6-8e8ba8232c85/volumes"
Sep 16 02:38:51.212768 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.212725    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"]
Sep 16 02:38:51.213119 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213106    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="015a8baa-9aad-4484-b6a0-b266b3842e50" containerName="maas-api"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213120    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="015a8baa-9aad-4484-b6a0-b266b3842e50" containerName="maas-api"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213133    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f8e17e-7c2a-4415-b101-a8357bd0ac33" containerName="maas-api"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213138    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f8e17e-7c2a-4415-b101-a8357bd0ac33" containerName="maas-api"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213153    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55adf607-21aa-40ef-b1e6-8e8ba8232c85" containerName="main"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213158    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="55adf607-21aa-40ef-b1e6-8e8ba8232c85" containerName="main"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213169    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9514d97e-2329-4757-aa9c-eb06ca82b309" containerName="main"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213175    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9514d97e-2329-4757-aa9c-eb06ca82b309" containerName="main"
Sep 16 02:38:51.213184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213186    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" containerName="main"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213192    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" containerName="main"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213198    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5642889a-1592-4949-9fcd-215e3656111b" containerName="maas-api"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213203    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5642889a-1592-4949-9fcd-215e3656111b" containerName="maas-api"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213270    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f8e17e-7c2a-4415-b101-a8357bd0ac33" containerName="maas-api"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213279    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5642889a-1592-4949-9fcd-215e3656111b" containerName="maas-api"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213287    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="015a8baa-9aad-4484-b6a0-b266b3842e50" containerName="maas-api"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213293    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9514d97e-2329-4757-aa9c-eb06ca82b309" containerName="main"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213302    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebcd016-962e-45c6-a0a9-5d1d909cb1cb" containerName="main"
Sep 16 02:38:51.213598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.213307    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="55adf607-21aa-40ef-b1e6-8e8ba8232c85" containerName="main"
Sep 16 02:38:51.218007 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.217979    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.220249 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.220227    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-efdec406-serving-cert\""
Sep 16 02:38:51.229238 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.229207    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"]
Sep 16 02:38:51.238139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.237873    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.238139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.237953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-metrics-certs\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.238139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.238018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j246r\" (UniqueName: \"kubernetes.io/projected/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-kube-api-access-j246r\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.238139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.238099    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-openshift-service-ca\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.339253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.339206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-openshift-service-ca\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.339464 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.339273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.339464 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.339310    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-metrics-certs\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.339464 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.339360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j246r\" (UniqueName: \"kubernetes.io/projected/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-kube-api-access-j246r\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.339464 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:38:51.339451    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-serving-cert: secret "maas-api-e2e-mt-efdec406-serving-cert" not found
Sep 16 02:38:51.339635 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:38:51.339519    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls podName:e6186d13-955b-4e24-8c64-b3e3fcc17c9f nodeName:}" failed. No retries permitted until 2026-09-16 02:38:51.839502776 +0000 UTC m=+1231.269564143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls") pod "maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7" (UID: "e6186d13-955b-4e24-8c64-b3e3fcc17c9f") : secret "maas-api-e2e-mt-efdec406-serving-cert" not found
Sep 16 02:38:51.339988 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.339954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-openshift-service-ca\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.341837 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.341812    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-metrics-certs\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.350000 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.349975    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j246r\" (UniqueName: \"kubernetes.io/projected/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-kube-api-access-j246r\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.843723 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.843683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:51.846220 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:51.846184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls\") pod \"maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:52.131837 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:52.131751    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:52.400900 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:52.400873    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"]
Sep 16 02:38:52.403753 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:38:52.403723    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6186d13_955b_4e24_8c64_b3e3fcc17c9f.slice/crio-1f4533a3d99dd34ed06bf61271f72764a68495ba70f6fa245bc6751e9a109d96 WatchSource:0}: Error finding container 1f4533a3d99dd34ed06bf61271f72764a68495ba70f6fa245bc6751e9a109d96: Status 404 returned error can't find the container with id 1f4533a3d99dd34ed06bf61271f72764a68495ba70f6fa245bc6751e9a109d96
Sep 16 02:38:52.417369 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:52.417339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7" event={"ID":"e6186d13-955b-4e24-8c64-b3e3fcc17c9f","Type":"ContainerStarted","Data":"1f4533a3d99dd34ed06bf61271f72764a68495ba70f6fa245bc6751e9a109d96"}
Sep 16 02:38:53.424201 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:53.424157    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7" event={"ID":"e6186d13-955b-4e24-8c64-b3e3fcc17c9f","Type":"ContainerStarted","Data":"ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6"}
Sep 16 02:38:53.424687 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:53.424265    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:38:53.439470 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:53.439403    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7" podStartSLOduration=2.07017768 podStartE2EDuration="2.439385578s" totalImagesPullingTime="369.207898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:38:53.437954738 +0000 UTC m=+1232.868016125" watchObservedRunningTime="2026-09-16 02:38:53.439385578 +0000 UTC m=+1232.869446963"
Sep 16 02:38:59.434865 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:38:59.434831    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:39:25.108074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:25.108036    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:39:32.269940 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.269896    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"]
Sep 16 02:39:32.270420 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.270147    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7" podUID="e6186d13-955b-4e24-8c64-b3e3fcc17c9f" containerName="maas-api" containerID="cri-o://ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6" gracePeriod=30
Sep 16 02:39:32.527013 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.526951    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:39:32.626172 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.626139    2556 generic.go:356] "Generic (PLEG): container finished" podID="e6186d13-955b-4e24-8c64-b3e3fcc17c9f" containerID="ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6" exitCode=0
Sep 16 02:39:32.626336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.626176    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"
Sep 16 02:39:32.626336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.626179    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7" event={"ID":"e6186d13-955b-4e24-8c64-b3e3fcc17c9f","Type":"ContainerDied","Data":"ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6"}
Sep 16 02:39:32.626336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.626256    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7" event={"ID":"e6186d13-955b-4e24-8c64-b3e3fcc17c9f","Type":"ContainerDied","Data":"1f4533a3d99dd34ed06bf61271f72764a68495ba70f6fa245bc6751e9a109d96"}
Sep 16 02:39:32.626336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.626280    2556 scope.go:117] "RemoveContainer" containerID="ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6"
Sep 16 02:39:32.635267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.635238    2556 scope.go:117] "RemoveContainer" containerID="ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6"
Sep 16 02:39:32.635569 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:39:32.635542    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6\": container with ID starting with ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6 not found: ID does not exist" containerID="ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6"
Sep 16 02:39:32.635642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.635572    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6"} err="failed to get container status \"ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6\": rpc error: code = NotFound desc = could not find container \"ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6\": container with ID starting with ee72ae16c9b659cd6aff9d09141b220ee81d00a54bffe8724ed705acb1e9caa6 not found: ID does not exist"
Sep 16 02:39:32.723320 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.723288    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-metrics-certs\") pod \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") "
Sep 16 02:39:32.723489 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.723360    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j246r\" (UniqueName: \"kubernetes.io/projected/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-kube-api-access-j246r\") pod \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") "
Sep 16 02:39:32.723489 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.723407    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-openshift-service-ca\") pod \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") "
Sep 16 02:39:32.723612 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.723552    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls\") pod \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\" (UID: \"e6186d13-955b-4e24-8c64-b3e3fcc17c9f\") "
Sep 16 02:39:32.723715 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.723694    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6186d13-955b-4e24-8c64-b3e3fcc17c9f" (UID: "e6186d13-955b-4e24-8c64-b3e3fcc17c9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:39:32.723876 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.723858    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:39:32.725520 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.725486    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6186d13-955b-4e24-8c64-b3e3fcc17c9f" (UID: "e6186d13-955b-4e24-8c64-b3e3fcc17c9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:39:32.725637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.725581    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-kube-api-access-j246r" (OuterVolumeSpecName: "kube-api-access-j246r") pod "e6186d13-955b-4e24-8c64-b3e3fcc17c9f" (UID: "e6186d13-955b-4e24-8c64-b3e3fcc17c9f"). InnerVolumeSpecName "kube-api-access-j246r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:32.725637 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.725593    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6186d13-955b-4e24-8c64-b3e3fcc17c9f" (UID: "e6186d13-955b-4e24-8c64-b3e3fcc17c9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:39:32.825267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.825200    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:39:32.825267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.825223    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:39:32.825267 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.825233    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j246r\" (UniqueName: \"kubernetes.io/projected/e6186d13-955b-4e24-8c64-b3e3fcc17c9f-kube-api-access-j246r\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:39:32.947368 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.947308    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"]
Sep 16 02:39:32.949252 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:32.949229    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efdec406-bcbdd8fd8-b6tf7"]
Sep 16 02:39:33.114335 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:33.114250    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6186d13-955b-4e24-8c64-b3e3fcc17c9f" path="/var/lib/kubelet/pods/e6186d13-955b-4e24-8c64-b3e3fcc17c9f/volumes"
Sep 16 02:39:50.935389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.935354    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9088-0bc710"]
Sep 16 02:39:50.935854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.935829    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6186d13-955b-4e24-8c64-b3e3fcc17c9f" containerName="maas-api"
Sep 16 02:39:50.935854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.935849    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6186d13-955b-4e24-8c64-b3e3fcc17c9f" containerName="maas-api"
Sep 16 02:39:50.935986 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.935918    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6186d13-955b-4e24-8c64-b3e3fcc17c9f" containerName="maas-api"
Sep 16 02:39:50.938819 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.938800    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:50.941143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.941118    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7nx29\""
Sep 16 02:39:50.943472 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.943446    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9088-0bc710"]
Sep 16 02:39:50.992878 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:50.992848    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q445\" (UniqueName: \"kubernetes.io/projected/3153d222-3377-408d-ac6d-41e735b79b35-kube-api-access-6q445\") pod \"test-curl-9088-0bc710\" (UID: \"3153d222-3377-408d-ac6d-41e735b79b35\") " pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:51.093646 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:51.093614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q445\" (UniqueName: \"kubernetes.io/projected/3153d222-3377-408d-ac6d-41e735b79b35-kube-api-access-6q445\") pod \"test-curl-9088-0bc710\" (UID: \"3153d222-3377-408d-ac6d-41e735b79b35\") " pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:51.101007 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:51.100983    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q445\" (UniqueName: \"kubernetes.io/projected/3153d222-3377-408d-ac6d-41e735b79b35-kube-api-access-6q445\") pod \"test-curl-9088-0bc710\" (UID: \"3153d222-3377-408d-ac6d-41e735b79b35\") " pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:51.249999 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:51.249923    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:51.519381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:51.519356    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9088-0bc710"]
Sep 16 02:39:51.521753 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:39:51.521723    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3153d222_3377_408d_ac6d_41e735b79b35.slice/crio-2520b31ed441ea018adbeb8fff643cf0d1f72817c2470d31a377f0beef798303 WatchSource:0}: Error finding container 2520b31ed441ea018adbeb8fff643cf0d1f72817c2470d31a377f0beef798303: Status 404 returned error can't find the container with id 2520b31ed441ea018adbeb8fff643cf0d1f72817c2470d31a377f0beef798303
Sep 16 02:39:51.723528 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:51.723496    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9088-0bc710" event={"ID":"3153d222-3377-408d-ac6d-41e735b79b35","Type":"ContainerStarted","Data":"00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d"}
Sep 16 02:39:51.723716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:51.723539    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9088-0bc710" event={"ID":"3153d222-3377-408d-ac6d-41e735b79b35","Type":"ContainerStarted","Data":"2520b31ed441ea018adbeb8fff643cf0d1f72817c2470d31a377f0beef798303"}
Sep 16 02:39:51.736713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:51.736667    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9088-0bc710" podStartSLOduration=1.736652516 podStartE2EDuration="1.736652516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:39:51.735317411 +0000 UTC m=+1291.165378793" watchObservedRunningTime="2026-09-16 02:39:51.736652516 +0000 UTC m=+1291.166713901"
Sep 16 02:39:52.304308 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:52.304277    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9088-0bc710"]
Sep 16 02:39:52.308369 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:52.308340    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9088-0bc710"]
Sep 16 02:39:52.728645 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:39:52.728613    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9088-0bc710" container="test-curl-9088-0bc710"
Sep 16 02:39:53.733034 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:53.732982    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9088-0bc710" podUID="3153d222-3377-408d-ac6d-41e735b79b35" containerName="test-curl-9088-0bc710" containerID="cri-o://00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d" gracePeriod=2
Sep 16 02:39:55.231255 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:55.231219    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"]
Sep 16 02:39:55.915537 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:55.915504    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:56.037582 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.037495    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q445\" (UniqueName: \"kubernetes.io/projected/3153d222-3377-408d-ac6d-41e735b79b35-kube-api-access-6q445\") pod \"3153d222-3377-408d-ac6d-41e735b79b35\" (UID: \"3153d222-3377-408d-ac6d-41e735b79b35\") "
Sep 16 02:39:56.039547 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.039522    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3153d222-3377-408d-ac6d-41e735b79b35-kube-api-access-6q445" (OuterVolumeSpecName: "kube-api-access-6q445") pod "3153d222-3377-408d-ac6d-41e735b79b35" (UID: "3153d222-3377-408d-ac6d-41e735b79b35"). InnerVolumeSpecName "kube-api-access-6q445". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:39:56.138875 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.138840    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q445\" (UniqueName: \"kubernetes.io/projected/3153d222-3377-408d-ac6d-41e735b79b35-kube-api-access-6q445\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:39:56.749018 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.748981    2556 generic.go:356] "Generic (PLEG): container finished" podID="3153d222-3377-408d-ac6d-41e735b79b35" containerID="00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d" exitCode=137
Sep 16 02:39:56.749529 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.749028    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:56.749529 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.749078    2556 scope.go:117] "RemoveContainer" containerID="00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d"
Sep 16 02:39:56.758087 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.758073    2556 scope.go:117] "RemoveContainer" containerID="00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d"
Sep 16 02:39:56.758355 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:39:56.758336    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d\": container with ID starting with 00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d not found: ID does not exist" containerID="00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d"
Sep 16 02:39:56.758452 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:56.758363    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d"} err="failed to get container status \"00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d\": rpc error: code = NotFound desc = could not find container \"00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d\": container with ID starting with 00d24114a06fb30fec2f7732223939f10edf085e2a270c8048c5e1a9d564eb6d not found: ID does not exist"
Sep 16 02:39:56.759080 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:39:56.759062    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9088-0bc710\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="3153d222-3377-408d-ac6d-41e735b79b35" pod="opendatahub/test-curl-9088-0bc710"
Sep 16 02:39:57.113979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.113901    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3153d222-3377-408d-ac6d-41e735b79b35" path="/var/lib/kubelet/pods/3153d222-3377-408d-ac6d-41e735b79b35/volumes"
Sep 16 02:39:57.897016 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.896980    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"]
Sep 16 02:39:57.897654 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.897632    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3153d222-3377-408d-ac6d-41e735b79b35" containerName="test-curl-9088-0bc710"
Sep 16 02:39:57.897654 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.897654    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3153d222-3377-408d-ac6d-41e735b79b35" containerName="test-curl-9088-0bc710"
Sep 16 02:39:57.897791 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.897753    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3153d222-3377-408d-ac6d-41e735b79b35" containerName="test-curl-9088-0bc710"
Sep 16 02:39:57.902787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.902765    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:57.906913 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.906071    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-fe9d17-serving-cert\""
Sep 16 02:39:57.907285 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.907229    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"]
Sep 16 02:39:57.957355 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.957318    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:57.957569 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.957369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7xpw\" (UniqueName: \"kubernetes.io/projected/e4c86d1b-f647-491c-bb73-0a75687b5641-kube-api-access-r7xpw\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:57.957569 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.957476    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:57.957569 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:57.957532    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c86d1b-f647-491c-bb73-0a75687b5641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.058047 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.058015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.058210 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.058074    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c86d1b-f647-491c-bb73-0a75687b5641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.058210 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.058133    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.058210 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.058157    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7xpw\" (UniqueName: \"kubernetes.io/projected/e4c86d1b-f647-491c-bb73-0a75687b5641-kube-api-access-r7xpw\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.058847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.058820    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c86d1b-f647-491c-bb73-0a75687b5641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.060534 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.060510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.060640 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.060540    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.065406 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.065383    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7xpw\" (UniqueName: \"kubernetes.io/projected/e4c86d1b-f647-491c-bb73-0a75687b5641-kube-api-access-r7xpw\") pod \"maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.219361 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.219321    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:58.492982 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.492947    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"]
Sep 16 02:39:58.495497 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:39:58.495459    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c86d1b_f647_491c_bb73_0a75687b5641.slice/crio-15a6f507628742800dcb65182de410fac1f1682eb0aef433ae780557173272ce WatchSource:0}: Error finding container 15a6f507628742800dcb65182de410fac1f1682eb0aef433ae780557173272ce: Status 404 returned error can't find the container with id 15a6f507628742800dcb65182de410fac1f1682eb0aef433ae780557173272ce
Sep 16 02:39:58.762005 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:58.761908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs" event={"ID":"e4c86d1b-f647-491c-bb73-0a75687b5641","Type":"ContainerStarted","Data":"15a6f507628742800dcb65182de410fac1f1682eb0aef433ae780557173272ce"}
Sep 16 02:39:59.768395 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:59.768359    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs" event={"ID":"e4c86d1b-f647-491c-bb73-0a75687b5641","Type":"ContainerStarted","Data":"9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305"}
Sep 16 02:39:59.768802 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:59.768483    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:39:59.786163 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:39:59.786117    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs" podStartSLOduration=2.3619710879999998 podStartE2EDuration="2.786104146s" totalImagesPullingTime="424.133058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:39:59.784263044 +0000 UTC m=+1299.214324433" watchObservedRunningTime="2026-09-16 02:39:59.786104146 +0000 UTC m=+1299.216165531"
Sep 16 02:40:05.777978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:05.777947    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:40:10.250574 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.250530    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4" podUID="64ed0f31-36ec-4c93-aa00-318e4b2f69a4" containerName="main" containerID="cri-o://e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e" gracePeriod=45
Sep 16 02:40:10.503814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.503748    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:40:10.683797 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.683761    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tmp-dir\") pod \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") "
Sep 16 02:40:10.683978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.683816    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-dshm\") pod \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") "
Sep 16 02:40:10.683978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.683860    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-model-cache\") pod \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") "
Sep 16 02:40:10.683978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.683887    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfxrf\" (UniqueName: \"kubernetes.io/projected/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-kube-api-access-pfxrf\") pod \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") "
Sep 16 02:40:10.683978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.683908    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-home\") pod \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") "
Sep 16 02:40:10.683978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.683961    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tls-certs\") pod \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\" (UID: \"64ed0f31-36ec-4c93-aa00-318e4b2f69a4\") "
Sep 16 02:40:10.684264 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.684120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64ed0f31-36ec-4c93-aa00-318e4b2f69a4" (UID: "64ed0f31-36ec-4c93-aa00-318e4b2f69a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:10.684264 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.684132    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "64ed0f31-36ec-4c93-aa00-318e4b2f69a4" (UID: "64ed0f31-36ec-4c93-aa00-318e4b2f69a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:10.684264 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.684239    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-home" (OuterVolumeSpecName: "home") pod "64ed0f31-36ec-4c93-aa00-318e4b2f69a4" (UID: "64ed0f31-36ec-4c93-aa00-318e4b2f69a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:10.684457 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.684366    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:10.684457 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.684386    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:10.684457 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.684400    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:10.686165 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.686141    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-dshm" (OuterVolumeSpecName: "dshm") pod "64ed0f31-36ec-4c93-aa00-318e4b2f69a4" (UID: "64ed0f31-36ec-4c93-aa00-318e4b2f69a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:40:10.686283 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.686201    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-kube-api-access-pfxrf" (OuterVolumeSpecName: "kube-api-access-pfxrf") pod "64ed0f31-36ec-4c93-aa00-318e4b2f69a4" (UID: "64ed0f31-36ec-4c93-aa00-318e4b2f69a4"). InnerVolumeSpecName "kube-api-access-pfxrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:10.686283 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.686267    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64ed0f31-36ec-4c93-aa00-318e4b2f69a4" (UID: "64ed0f31-36ec-4c93-aa00-318e4b2f69a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:40:10.785535 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.785461    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxrf\" (UniqueName: \"kubernetes.io/projected/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-kube-api-access-pfxrf\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:10.785535 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.785489    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:10.785535 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.785503    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ed0f31-36ec-4c93-aa00-318e4b2f69a4-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:10.826021 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.825993    2556 generic.go:356] "Generic (PLEG): container finished" podID="64ed0f31-36ec-4c93-aa00-318e4b2f69a4" containerID="e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e" exitCode=0
Sep 16 02:40:10.826113 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.826037    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"
Sep 16 02:40:10.826113 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.826081    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4" event={"ID":"64ed0f31-36ec-4c93-aa00-318e4b2f69a4","Type":"ContainerDied","Data":"e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e"}
Sep 16 02:40:10.826193 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.826119    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4" event={"ID":"64ed0f31-36ec-4c93-aa00-318e4b2f69a4","Type":"ContainerDied","Data":"765f9f2815a7d3dfcd01c30c977bee5d90bd64a0e7709a97b067bb9ce324beb3"}
Sep 16 02:40:10.826193 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.826138    2556 scope.go:117] "RemoveContainer" containerID="e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e"
Sep 16 02:40:10.834608 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.834593    2556 scope.go:117] "RemoveContainer" containerID="e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e"
Sep 16 02:40:10.834863 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:10.834846    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e\": container with ID starting with e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e not found: ID does not exist" containerID="e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e"
Sep 16 02:40:10.834904 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.834872    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e"} err="failed to get container status \"e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e\": rpc error: code = NotFound desc = could not find container \"e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e\": container with ID starting with e1b30b8251cb775c7ee2d6a7cf9c8c3198a81ad5981c0a6b5078c29d11be896e not found: ID does not exist"
Sep 16 02:40:10.844823 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.844785    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"]
Sep 16 02:40:10.846793 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:10.846771    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/auth-test-model-eee708-kserve-65478b77c9-5phl4"]
Sep 16 02:40:11.114923 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:11.114845    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ed0f31-36ec-4c93-aa00-318e4b2f69a4" path="/var/lib/kubelet/pods/64ed0f31-36ec-4c93-aa00-318e4b2f69a4/volumes"
Sep 16 02:40:25.905252 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:25.905064    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"]
Sep 16 02:40:25.905781 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:25.905699    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ed0f31-36ec-4c93-aa00-318e4b2f69a4" containerName="main"
Sep 16 02:40:25.905781 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:25.905714    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ed0f31-36ec-4c93-aa00-318e4b2f69a4" containerName="main"
Sep 16 02:40:25.905911 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:25.905811    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ed0f31-36ec-4c93-aa00-318e4b2f69a4" containerName="main"
Sep 16 02:40:25.911055 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:25.911032    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:25.913968 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:25.913733    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e86d9e-serving-cert\""
Sep 16 02:40:25.915008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:25.914987    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"]
Sep 16 02:40:26.008107 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.008076    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.008270 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.008160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13ee01d-49cb-4d16-aa5a-e7313245935a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.008270 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.008237    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.008351 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.008315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5gv\" (UniqueName: \"kubernetes.io/projected/c13ee01d-49cb-4d16-aa5a-e7313245935a-kube-api-access-sn5gv\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.108773 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.108740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13ee01d-49cb-4d16-aa5a-e7313245935a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.108937 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.108801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.108937 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.108877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5gv\" (UniqueName: \"kubernetes.io/projected/c13ee01d-49cb-4d16-aa5a-e7313245935a-kube-api-access-sn5gv\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.108937 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.108908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.108937 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:26.108914    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-serving-cert: secret "maas-api-e2e-shared-b-w1-e86d9e-serving-cert" not found
Sep 16 02:40:26.109163 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:26.108989    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls podName:c13ee01d-49cb-4d16-aa5a-e7313245935a nodeName:}" failed. No retries permitted until 2026-09-16 02:40:26.608967305 +0000 UTC m=+1326.039028675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls") pod "maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr" (UID: "c13ee01d-49cb-4d16-aa5a-e7313245935a") : secret "maas-api-e2e-shared-b-w1-e86d9e-serving-cert" not found
Sep 16 02:40:26.109413 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.109386    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13ee01d-49cb-4d16-aa5a-e7313245935a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.111293 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.111276    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.119372 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.119347    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5gv\" (UniqueName: \"kubernetes.io/projected/c13ee01d-49cb-4d16-aa5a-e7313245935a-kube-api-access-sn5gv\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.613268 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.613230    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.615649 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.615620    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:26.826406 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:26.826368    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:27.098227 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:27.098206    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"]
Sep 16 02:40:27.101043 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:40:27.101016    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13ee01d_49cb_4d16_aa5a_e7313245935a.slice/crio-0dd836f053dd5310b76faf441526e99e464dcfe0fa70277633f188b70fe84127 WatchSource:0}: Error finding container 0dd836f053dd5310b76faf441526e99e464dcfe0fa70277633f188b70fe84127: Status 404 returned error can't find the container with id 0dd836f053dd5310b76faf441526e99e464dcfe0fa70277633f188b70fe84127
Sep 16 02:40:27.102755 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:27.102727    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:40:27.922181 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:27.922145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr" event={"ID":"c13ee01d-49cb-4d16-aa5a-e7313245935a","Type":"ContainerStarted","Data":"e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943"}
Sep 16 02:40:27.922381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:27.922198    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr" event={"ID":"c13ee01d-49cb-4d16-aa5a-e7313245935a","Type":"ContainerStarted","Data":"0dd836f053dd5310b76faf441526e99e464dcfe0fa70277633f188b70fe84127"}
Sep 16 02:40:27.922381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:27.922273    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:27.938143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:27.938098    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr" podStartSLOduration=2.6244103880000003 podStartE2EDuration="2.938083595s" totalImagesPullingTime="313.673207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:27.936711565 +0000 UTC m=+1327.366772952" watchObservedRunningTime="2026-09-16 02:40:27.938083595 +0000 UTC m=+1327.368145012"
Sep 16 02:40:33.934088 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:33.934056    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:40:35.114315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:35.114277    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:40:50.071673 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.071593    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9085-ba1e3c"]
Sep 16 02:40:50.075085 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.075065    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:50.077324 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.077305    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7nx29\""
Sep 16 02:40:50.081102 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.081069    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-ba1e3c"]
Sep 16 02:40:50.128824 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.128785    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5vl\" (UniqueName: \"kubernetes.io/projected/26b57352-35a6-4c46-8fd6-5cbe18feff16-kube-api-access-jz5vl\") pod \"test-curl-9085-ba1e3c\" (UID: \"26b57352-35a6-4c46-8fd6-5cbe18feff16\") " pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:50.230008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.229962    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5vl\" (UniqueName: \"kubernetes.io/projected/26b57352-35a6-4c46-8fd6-5cbe18feff16-kube-api-access-jz5vl\") pod \"test-curl-9085-ba1e3c\" (UID: \"26b57352-35a6-4c46-8fd6-5cbe18feff16\") " pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:50.241877 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.241842    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5vl\" (UniqueName: \"kubernetes.io/projected/26b57352-35a6-4c46-8fd6-5cbe18feff16-kube-api-access-jz5vl\") pod \"test-curl-9085-ba1e3c\" (UID: \"26b57352-35a6-4c46-8fd6-5cbe18feff16\") " pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:50.387047 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.386959    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:50.659418 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:50.659394    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-ba1e3c"]
Sep 16 02:40:50.661285 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:40:50.661256    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26b57352_35a6_4c46_8fd6_5cbe18feff16.slice/crio-4abc2467540424d2c77fff2e00b617262de60d158e55742b369a08a1d366c009 WatchSource:0}: Error finding container 4abc2467540424d2c77fff2e00b617262de60d158e55742b369a08a1d366c009: Status 404 returned error can't find the container with id 4abc2467540424d2c77fff2e00b617262de60d158e55742b369a08a1d366c009
Sep 16 02:40:51.039069 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.039037    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-ba1e3c" event={"ID":"26b57352-35a6-4c46-8fd6-5cbe18feff16","Type":"ContainerStarted","Data":"4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea"}
Sep 16 02:40:51.039244 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.039076    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-ba1e3c" event={"ID":"26b57352-35a6-4c46-8fd6-5cbe18feff16","Type":"ContainerStarted","Data":"4abc2467540424d2c77fff2e00b617262de60d158e55742b369a08a1d366c009"}
Sep 16 02:40:51.053278 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.053227    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9085-ba1e3c" podStartSLOduration=1.053210445 podStartE2EDuration="1.053210445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:51.051405173 +0000 UTC m=+1350.481466560" watchObservedRunningTime="2026-09-16 02:40:51.053210445 +0000 UTC m=+1350.483271828"
Sep 16 02:40:51.614987 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.614943    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9085-ba1e3c"]
Sep 16 02:40:51.618529 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.618504    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9085-ba1e3c"]
Sep 16 02:40:51.732513 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.732479    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9085-6ee1fd"]
Sep 16 02:40:51.732890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.732877    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" containerName="test-curl-9085-ba1e3c"
Sep 16 02:40:51.732938 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.732892    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" containerName="test-curl-9085-ba1e3c"
Sep 16 02:40:51.732974 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.732966    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" containerName="test-curl-9085-ba1e3c"
Sep 16 02:40:51.735954 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.735938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:51.741499 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.741458    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-6ee1fd"]
Sep 16 02:40:51.845082 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.845051    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnsbs\" (UniqueName: \"kubernetes.io/projected/ba3488c6-751c-421c-8d8b-043fa24c0589-kube-api-access-bnsbs\") pod \"test-curl-9085-6ee1fd\" (UID: \"ba3488c6-751c-421c-8d8b-043fa24c0589\") " pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:51.946086 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.946045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnsbs\" (UniqueName: \"kubernetes.io/projected/ba3488c6-751c-421c-8d8b-043fa24c0589-kube-api-access-bnsbs\") pod \"test-curl-9085-6ee1fd\" (UID: \"ba3488c6-751c-421c-8d8b-043fa24c0589\") " pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:51.953019 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:51.952987    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnsbs\" (UniqueName: \"kubernetes.io/projected/ba3488c6-751c-421c-8d8b-043fa24c0589-kube-api-access-bnsbs\") pod \"test-curl-9085-6ee1fd\" (UID: \"ba3488c6-751c-421c-8d8b-043fa24c0589\") " pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:52.045026 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:52.044937    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9085-ba1e3c" container="test-curl-9085-ba1e3c"
Sep 16 02:40:52.050867 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:52.050834    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:52.320763 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:52.320737    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-6ee1fd"]
Sep 16 02:40:52.323170 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:40:52.323137    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba3488c6_751c_421c_8d8b_043fa24c0589.slice/crio-a9fcf3c82ce75ea6b846c1b57d67c4d6ec2856fdd5ae87b23d8c44cc824b6b6c WatchSource:0}: Error finding container a9fcf3c82ce75ea6b846c1b57d67c4d6ec2856fdd5ae87b23d8c44cc824b6b6c: Status 404 returned error can't find the container with id a9fcf3c82ce75ea6b846c1b57d67c4d6ec2856fdd5ae87b23d8c44cc824b6b6c
Sep 16 02:40:53.052000 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.051959    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-6ee1fd" event={"ID":"ba3488c6-751c-421c-8d8b-043fa24c0589","Type":"ContainerStarted","Data":"7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8"}
Sep 16 02:40:53.052000 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.052014    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-6ee1fd" event={"ID":"ba3488c6-751c-421c-8d8b-043fa24c0589","Type":"ContainerStarted","Data":"a9fcf3c82ce75ea6b846c1b57d67c4d6ec2856fdd5ae87b23d8c44cc824b6b6c"}
Sep 16 02:40:53.052511 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.052056    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9085-ba1e3c" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" containerName="test-curl-9085-ba1e3c" containerID="cri-o://4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea" gracePeriod=2
Sep 16 02:40:53.054139 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:53.054107    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ba1e3c\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:53.066599 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:53.066564    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ba1e3c\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:53.067176 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.067141    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9085-6ee1fd" podStartSLOduration=2.067116896 podStartE2EDuration="2.067116896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:53.064900567 +0000 UTC m=+1352.494961953" watchObservedRunningTime="2026-09-16 02:40:53.067116896 +0000 UTC m=+1352.497178281"
Sep 16 02:40:53.650719 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.650675    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9085-6ee1fd"]
Sep 16 02:40:53.652807 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.652777    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9085-6ee1fd"]
Sep 16 02:40:53.768958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.768878    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9085-d5acce"]
Sep 16 02:40:53.769300 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.769287    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" containerName="test-curl-9085-6ee1fd"
Sep 16 02:40:53.769345 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.769301    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" containerName="test-curl-9085-6ee1fd"
Sep 16 02:40:53.769378 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.769373    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" containerName="test-curl-9085-6ee1fd"
Sep 16 02:40:53.772365 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.772348    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:40:53.777845 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.777817    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-d5acce"]
Sep 16 02:40:53.868102 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.868066    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mqf\" (UniqueName: \"kubernetes.io/projected/491557ce-730a-4247-909e-2a26782115bb-kube-api-access-n8mqf\") pod \"test-curl-9085-d5acce\" (UID: \"491557ce-730a-4247-909e-2a26782115bb\") " pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:40:53.969253 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.969208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mqf\" (UniqueName: \"kubernetes.io/projected/491557ce-730a-4247-909e-2a26782115bb-kube-api-access-n8mqf\") pod \"test-curl-9085-d5acce\" (UID: \"491557ce-730a-4247-909e-2a26782115bb\") " pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:40:53.976138 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:53.976114    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mqf\" (UniqueName: \"kubernetes.io/projected/491557ce-730a-4247-909e-2a26782115bb-kube-api-access-n8mqf\") pod \"test-curl-9085-d5acce\" (UID: \"491557ce-730a-4247-909e-2a26782115bb\") " pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:40:54.057104 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:54.057032    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9085-6ee1fd" container="test-curl-9085-6ee1fd"
Sep 16 02:40:54.084131 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:54.084109    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:40:54.356958 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:54.356929    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-d5acce"]
Sep 16 02:40:54.358701 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:40:54.358668    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod491557ce_730a_4247_909e_2a26782115bb.slice/crio-c05766b120dc8a35f112239274cc5b47ba0a9d07ed535a76e188c55b3ba8130d WatchSource:0}: Error finding container c05766b120dc8a35f112239274cc5b47ba0a9d07ed535a76e188c55b3ba8130d: Status 404 returned error can't find the container with id c05766b120dc8a35f112239274cc5b47ba0a9d07ed535a76e188c55b3ba8130d
Sep 16 02:40:55.065420 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.065389    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-d5acce" event={"ID":"491557ce-730a-4247-909e-2a26782115bb","Type":"ContainerStarted","Data":"e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941"}
Sep 16 02:40:55.065810 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.065452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-d5acce" event={"ID":"491557ce-730a-4247-909e-2a26782115bb","Type":"ContainerStarted","Data":"c05766b120dc8a35f112239274cc5b47ba0a9d07ed535a76e188c55b3ba8130d"}
Sep 16 02:40:55.065810 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.065594    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9085-6ee1fd" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" containerName="test-curl-9085-6ee1fd" containerID="cri-o://7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8" gracePeriod=2
Sep 16 02:40:55.078208 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.078167    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9085-d5acce" podStartSLOduration=2.078151475 podStartE2EDuration="2.078151475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:55.077490399 +0000 UTC m=+1354.507551786" watchObservedRunningTime="2026-09-16 02:40:55.078151475 +0000 UTC m=+1354.508212861"
Sep 16 02:40:55.079309 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:55.079279    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-6ee1fd\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:55.243391 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.243362    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:55.381994 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.381903    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz5vl\" (UniqueName: \"kubernetes.io/projected/26b57352-35a6-4c46-8fd6-5cbe18feff16-kube-api-access-jz5vl\") pod \"26b57352-35a6-4c46-8fd6-5cbe18feff16\" (UID: \"26b57352-35a6-4c46-8fd6-5cbe18feff16\") "
Sep 16 02:40:55.383978 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.383955    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b57352-35a6-4c46-8fd6-5cbe18feff16-kube-api-access-jz5vl" (OuterVolumeSpecName: "kube-api-access-jz5vl") pod "26b57352-35a6-4c46-8fd6-5cbe18feff16" (UID: "26b57352-35a6-4c46-8fd6-5cbe18feff16"). InnerVolumeSpecName "kube-api-access-jz5vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:55.482630 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.482602    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz5vl\" (UniqueName: \"kubernetes.io/projected/26b57352-35a6-4c46-8fd6-5cbe18feff16-kube-api-access-jz5vl\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:55.623443 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.623386    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9085-d5acce"]
Sep 16 02:40:55.627015 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.626984    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9085-d5acce"]
Sep 16 02:40:55.738536 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.738503    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9085-ad7341"]
Sep 16 02:40:55.738905 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.738893    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491557ce-730a-4247-909e-2a26782115bb" containerName="test-curl-9085-d5acce"
Sep 16 02:40:55.738951 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.738906    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="491557ce-730a-4247-909e-2a26782115bb" containerName="test-curl-9085-d5acce"
Sep 16 02:40:55.738983 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.738970    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="491557ce-730a-4247-909e-2a26782115bb" containerName="test-curl-9085-d5acce"
Sep 16 02:40:55.745302 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.742614    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:40:55.751332 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.751307    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-ad7341"]
Sep 16 02:40:55.886260 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.886223    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnxd\" (UniqueName: \"kubernetes.io/projected/2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23-kube-api-access-cpnxd\") pod \"test-curl-9085-ad7341\" (UID: \"2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23\") " pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:40:55.987610 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.987572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnxd\" (UniqueName: \"kubernetes.io/projected/2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23-kube-api-access-cpnxd\") pod \"test-curl-9085-ad7341\" (UID: \"2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23\") " pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:40:55.994837 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:55.994776    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnxd\" (UniqueName: \"kubernetes.io/projected/2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23-kube-api-access-cpnxd\") pod \"test-curl-9085-ad7341\" (UID: \"2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23\") " pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:40:56.056877 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:56.056846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:40:56.072885 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:56.072857    2556 generic.go:356] "Generic (PLEG): container finished" podID="26b57352-35a6-4c46-8fd6-5cbe18feff16" containerID="4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea" exitCode=137
Sep 16 02:40:56.073260 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:56.072904    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:56.073260 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:56.072956    2556 scope.go:117] "RemoveContainer" containerID="4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea"
Sep 16 02:40:56.073466 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:56.073409    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9085-d5acce" container="test-curl-9085-d5acce"
Sep 16 02:40:56.085568 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:56.085472    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ba1e3c\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:56.085634 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:56.085602    2556 scope.go:117] "RemoveContainer" containerID="4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea"
Sep 16 02:40:56.085916 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:56.085894    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea\": container with ID starting with 4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea not found: ID does not exist" containerID="4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea"
Sep 16 02:40:56.086000 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:56.085929    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea"} err="failed to get container status \"4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea\": rpc error: code = NotFound desc = could not find container \"4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea\": container with ID starting with 4f7cb6a10596e5d6987331ed0b40a3b1ebbf59124fc5f00a140b48c1daeaf5ea not found: ID does not exist"
Sep 16 02:40:56.325085 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:56.325054    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9085-ad7341"]
Sep 16 02:40:56.327405 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:40:56.327376    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec4a6da_9aa8_4d03_acdb_cd1fcda1fb23.slice/crio-dc1ee3c6459ced29053fc9651058584cafe7b4d75e711ed8ada74aba8316d4cb WatchSource:0}: Error finding container dc1ee3c6459ced29053fc9651058584cafe7b4d75e711ed8ada74aba8316d4cb: Status 404 returned error can't find the container with id dc1ee3c6459ced29053fc9651058584cafe7b4d75e711ed8ada74aba8316d4cb
Sep 16 02:40:57.081707 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.081680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-ad7341" event={"ID":"2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23","Type":"ContainerStarted","Data":"18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49"}
Sep 16 02:40:57.081991 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.081725    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9085-ad7341" event={"ID":"2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23","Type":"ContainerStarted","Data":"dc1ee3c6459ced29053fc9651058584cafe7b4d75e711ed8ada74aba8316d4cb"}
Sep 16 02:40:57.081991 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.081938    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9085-d5acce" podUID="491557ce-730a-4247-909e-2a26782115bb" containerName="test-curl-9085-d5acce" containerID="cri-o://e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941" gracePeriod=2
Sep 16 02:40:57.083539 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:57.083505    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ba1e3c\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:57.095934 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.095896    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9085-ad7341" podStartSLOduration=2.095881288 podStartE2EDuration="2.095881288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:40:57.094886177 +0000 UTC m=+1356.524947566" watchObservedRunningTime="2026-09-16 02:40:57.095881288 +0000 UTC m=+1356.525942673"
Sep 16 02:40:57.096606 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:57.096581    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ba1e3c\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" pod="opendatahub/test-curl-9085-ba1e3c"
Sep 16 02:40:57.098110 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:57.098088    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-d5acce\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="491557ce-730a-4247-909e-2a26782115bb" pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:40:57.113639 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.113616    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b57352-35a6-4c46-8fd6-5cbe18feff16" path="/var/lib/kubelet/pods/26b57352-35a6-4c46-8fd6-5cbe18feff16/volumes"
Sep 16 02:40:57.204329 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.204308    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:57.300242 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.300217    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnsbs\" (UniqueName: \"kubernetes.io/projected/ba3488c6-751c-421c-8d8b-043fa24c0589-kube-api-access-bnsbs\") pod \"ba3488c6-751c-421c-8d8b-043fa24c0589\" (UID: \"ba3488c6-751c-421c-8d8b-043fa24c0589\") "
Sep 16 02:40:57.302137 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.302111    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3488c6-751c-421c-8d8b-043fa24c0589-kube-api-access-bnsbs" (OuterVolumeSpecName: "kube-api-access-bnsbs") pod "ba3488c6-751c-421c-8d8b-043fa24c0589" (UID: "ba3488c6-751c-421c-8d8b-043fa24c0589"). InnerVolumeSpecName "kube-api-access-bnsbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:57.401160 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.401090    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnsbs\" (UniqueName: \"kubernetes.io/projected/ba3488c6-751c-421c-8d8b-043fa24c0589-kube-api-access-bnsbs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:40:57.639308 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.639236    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9085-ad7341"]
Sep 16 02:40:57.641832 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:57.641801    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9085-ad7341"]
Sep 16 02:40:58.088462 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:58.088413    2556 generic.go:356] "Generic (PLEG): container finished" podID="ba3488c6-751c-421c-8d8b-043fa24c0589" containerID="7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8" exitCode=137
Sep 16 02:40:58.088968 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:58.088471    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:58.088968 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:58.088478    2556 scope.go:117] "RemoveContainer" containerID="7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8"
Sep 16 02:40:58.088968 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:58.088739    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9085-ad7341" container="test-curl-9085-ad7341"
Sep 16 02:40:58.097190 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:58.097172    2556 scope.go:117] "RemoveContainer" containerID="7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8"
Sep 16 02:40:58.097473 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:58.097454    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8\": container with ID starting with 7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8 not found: ID does not exist" containerID="7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8"
Sep 16 02:40:58.097534 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:58.097482    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8"} err="failed to get container status \"7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8\": rpc error: code = NotFound desc = could not find container \"7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8\": container with ID starting with 7efb4e69105578a737fc1a3bcad073ef9109014eaa626e0bc8a2f5023ed639d8 not found: ID does not exist"
Sep 16 02:40:58.098581 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:58.098560    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-6ee1fd\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:59.095078 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:59.095050    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9085-ad7341" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" containerName="test-curl-9085-ad7341" containerID="cri-o://18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49" gracePeriod=2
Sep 16 02:40:59.096968 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:59.096935    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ad7341\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:40:59.098544 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:40:59.098513    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-6ee1fd\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" pod="opendatahub/test-curl-9085-6ee1fd"
Sep 16 02:40:59.113192 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:59.113168    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3488c6-751c-421c-8d8b-043fa24c0589" path="/var/lib/kubelet/pods/ba3488c6-751c-421c-8d8b-043fa24c0589/volumes"
Sep 16 02:40:59.271106 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:59.271082    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:40:59.419673 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:59.419651    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8mqf\" (UniqueName: \"kubernetes.io/projected/491557ce-730a-4247-909e-2a26782115bb-kube-api-access-n8mqf\") pod \"491557ce-730a-4247-909e-2a26782115bb\" (UID: \"491557ce-730a-4247-909e-2a26782115bb\") "
Sep 16 02:40:59.421687 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:59.421660    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491557ce-730a-4247-909e-2a26782115bb-kube-api-access-n8mqf" (OuterVolumeSpecName: "kube-api-access-n8mqf") pod "491557ce-730a-4247-909e-2a26782115bb" (UID: "491557ce-730a-4247-909e-2a26782115bb"). InnerVolumeSpecName "kube-api-access-n8mqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:40:59.521040 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:40:59.520998    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mqf\" (UniqueName: \"kubernetes.io/projected/491557ce-730a-4247-909e-2a26782115bb-kube-api-access-n8mqf\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:41:00.102381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:00.102348    2556 generic.go:356] "Generic (PLEG): container finished" podID="491557ce-730a-4247-909e-2a26782115bb" containerID="e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941" exitCode=137
Sep 16 02:41:00.102816 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:00.102391    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:41:00.102816 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:00.102401    2556 scope.go:117] "RemoveContainer" containerID="e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941"
Sep 16 02:41:00.111049 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:00.111027    2556 scope.go:117] "RemoveContainer" containerID="e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941"
Sep 16 02:41:00.111314 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:41:00.111297    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941\": container with ID starting with e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941 not found: ID does not exist" containerID="e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941"
Sep 16 02:41:00.111361 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:00.111321    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941"} err="failed to get container status \"e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941\": rpc error: code = NotFound desc = could not find container \"e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941\": container with ID starting with e8f0fe49a321b532c44f2a8e8bd86e067cc066b2ff80b4b0b325be05e1794941 not found: ID does not exist"
Sep 16 02:41:00.112042 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:41:00.112021    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-d5acce\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="491557ce-730a-4247-909e-2a26782115bb" pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:41:01.112565 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:41:01.112535    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-d5acce\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="491557ce-730a-4247-909e-2a26782115bb" pod="opendatahub/test-curl-9085-d5acce"
Sep 16 02:41:01.114835 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:01.114811    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491557ce-730a-4247-909e-2a26782115bb" path="/var/lib/kubelet/pods/491557ce-730a-4247-909e-2a26782115bb/volumes"
Sep 16 02:41:01.234307 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:01.234282    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:41:01.236368 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:41:01.236337    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ad7341\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:41:01.337562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:01.337520    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpnxd\" (UniqueName: \"kubernetes.io/projected/2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23-kube-api-access-cpnxd\") pod \"2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23\" (UID: \"2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23\") "
Sep 16 02:41:01.339568 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:01.339547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23-kube-api-access-cpnxd" (OuterVolumeSpecName: "kube-api-access-cpnxd") pod "2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" (UID: "2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23"). InnerVolumeSpecName "kube-api-access-cpnxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:41:01.438686 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:01.438655    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnxd\" (UniqueName: \"kubernetes.io/projected/2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23-kube-api-access-cpnxd\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:41:02.116076 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:02.116031    2556 generic.go:356] "Generic (PLEG): container finished" podID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" containerID="18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49" exitCode=137
Sep 16 02:41:02.116076 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:02.116067    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:41:02.116593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:02.116114    2556 scope.go:117] "RemoveContainer" containerID="18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49"
Sep 16 02:41:02.118006 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:41:02.117967    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ad7341\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:41:02.125786 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:02.125769    2556 scope.go:117] "RemoveContainer" containerID="18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49"
Sep 16 02:41:02.126052 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:41:02.126032    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49\": container with ID starting with 18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49 not found: ID does not exist" containerID="18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49"
Sep 16 02:41:02.126120 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:02.126057    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49"} err="failed to get container status \"18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49\": rpc error: code = NotFound desc = could not find container \"18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49\": container with ID starting with 18a44fe44adff01e69731a4d1ab96d6799644278030af85c5c8f55bc58111c49 not found: ID does not exist"
Sep 16 02:41:02.126898 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:41:02.126875    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9085-ad7341\" is forbidden: User \"system:node:ip-10-0-143-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-48.ec2.internal' and this object" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" pod="opendatahub/test-curl-9085-ad7341"
Sep 16 02:41:03.114337 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.114303    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" path="/var/lib/kubelet/pods/2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23/volumes"
Sep 16 02:41:03.904988 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.904952    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"]
Sep 16 02:41:03.905575 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.905555    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" containerName="test-curl-9085-ad7341"
Sep 16 02:41:03.905688 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.905578    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" containerName="test-curl-9085-ad7341"
Sep 16 02:41:03.905748 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.905711    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec4a6da-9aa8-4d03-acdb-cd1fcda1fb23" containerName="test-curl-9085-ad7341"
Sep 16 02:41:03.910273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.910245    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:03.912502 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.912475    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-73bfc1e9-serving-cert\""
Sep 16 02:41:03.926766 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:03.926740    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"]
Sep 16 02:41:04.061615 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.061579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72374-4bbd-48be-a9b4-417a17240a6b-openshift-service-ca\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.061615 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.061618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvd8\" (UniqueName: \"kubernetes.io/projected/01b72374-4bbd-48be-a9b4-417a17240a6b-kube-api-access-fpvd8\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.061881 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.061754    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-maas-api-tls\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.061881 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.061787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-metrics-certs\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.163351 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.163254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-maas-api-tls\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.163351 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.163293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-metrics-certs\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.163351 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.163344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72374-4bbd-48be-a9b4-417a17240a6b-openshift-service-ca\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.163694 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.163365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvd8\" (UniqueName: \"kubernetes.io/projected/01b72374-4bbd-48be-a9b4-417a17240a6b-kube-api-access-fpvd8\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.164128 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.164106    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72374-4bbd-48be-a9b4-417a17240a6b-openshift-service-ca\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.165725 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.165691    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-maas-api-tls\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.165993 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.165971    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-metrics-certs\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.171132 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.171112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvd8\" (UniqueName: \"kubernetes.io/projected/01b72374-4bbd-48be-a9b4-417a17240a6b-kube-api-access-fpvd8\") pod \"maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.228162 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.228129    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:04.504971 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:04.504943    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"]
Sep 16 02:41:04.506936 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:41:04.506911    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b72374_4bbd_48be_a9b4_417a17240a6b.slice/crio-1cc9b3685c4c3e0ae34dbd9be3fd1bf7d7576b756bf146b90bd7fea241037c43 WatchSource:0}: Error finding container 1cc9b3685c4c3e0ae34dbd9be3fd1bf7d7576b756bf146b90bd7fea241037c43: Status 404 returned error can't find the container with id 1cc9b3685c4c3e0ae34dbd9be3fd1bf7d7576b756bf146b90bd7fea241037c43
Sep 16 02:41:05.133791 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:05.133699    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p" event={"ID":"01b72374-4bbd-48be-a9b4-417a17240a6b","Type":"ContainerStarted","Data":"fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55"}
Sep 16 02:41:05.133791 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:05.133773    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:05.134252 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:05.133798    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p" event={"ID":"01b72374-4bbd-48be-a9b4-417a17240a6b","Type":"ContainerStarted","Data":"1cc9b3685c4c3e0ae34dbd9be3fd1bf7d7576b756bf146b90bd7fea241037c43"}
Sep 16 02:41:05.149760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:05.149700    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p" podStartSLOduration=1.779041721 podStartE2EDuration="2.149686218s" totalImagesPullingTime="370.644497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:41:05.148383181 +0000 UTC m=+1364.578444583" watchObservedRunningTime="2026-09-16 02:41:05.149686218 +0000 UTC m=+1364.579747600"
Sep 16 02:41:11.145440 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:11.145400    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:41:17.148910 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.148871    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"]
Sep 16 02:41:17.152248 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.152225    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.154556 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.154533    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"openshift-service-ca.crt\""
Sep 16 02:41:17.155360 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.155338    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"default-dockercfg-2w275\""
Sep 16 02:41:17.155512 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.155491    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"sub-test-model-eee708-kserve-self-signed-certs\""
Sep 16 02:41:17.155596 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.155493    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-eee708\"/\"kube-root-ca.crt\""
Sep 16 02:41:17.160131 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.160110    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"]
Sep 16 02:41:17.178683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.178659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e85920-c29c-4100-a5ab-c0d6f7c34256-tls-certs\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.178796 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.178696    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-dshm\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.178796 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.178763    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-home\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.178796 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.178787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-tmp-dir\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.178976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.178850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87v9\" (UniqueName: \"kubernetes.io/projected/88e85920-c29c-4100-a5ab-c0d6f7c34256-kube-api-access-m87v9\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.178976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.178899    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-model-cache\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.279683 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.279652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-tmp-dir\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.279829 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.279703    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87v9\" (UniqueName: \"kubernetes.io/projected/88e85920-c29c-4100-a5ab-c0d6f7c34256-kube-api-access-m87v9\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.279829 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.279744    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-model-cache\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.279829 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.279798    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e85920-c29c-4100-a5ab-c0d6f7c34256-tls-certs\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.279941 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.279836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-dshm\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.279941 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.279877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-home\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.280142 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.280108    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-tmp-dir\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.280259 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.280140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-model-cache\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.280259 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.280165    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-home\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.281989 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.281967    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-dshm\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.282265 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.282247    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e85920-c29c-4100-a5ab-c0d6f7c34256-tls-certs\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.286982 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.286959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87v9\" (UniqueName: \"kubernetes.io/projected/88e85920-c29c-4100-a5ab-c0d6f7c34256-kube-api-access-m87v9\") pod \"sub-test-model-eee708-kserve-fccf4c6dc-xcz5w\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") " pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.463627 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.463599    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:17.733908 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:17.733829    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"]
Sep 16 02:41:17.736869 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:41:17.736839    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e85920_c29c_4100_a5ab_c0d6f7c34256.slice/crio-547c6315251ed37c9d3b26694ec4c3b08662557079d281ce1bf781ac243d3580 WatchSource:0}: Error finding container 547c6315251ed37c9d3b26694ec4c3b08662557079d281ce1bf781ac243d3580: Status 404 returned error can't find the container with id 547c6315251ed37c9d3b26694ec4c3b08662557079d281ce1bf781ac243d3580
Sep 16 02:41:18.217490 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:18.217457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w" event={"ID":"88e85920-c29c-4100-a5ab-c0d6f7c34256","Type":"ContainerStarted","Data":"c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6"}
Sep 16 02:41:18.217867 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:18.217498    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w" event={"ID":"88e85920-c29c-4100-a5ab-c0d6f7c34256","Type":"ContainerStarted","Data":"547c6315251ed37c9d3b26694ec4c3b08662557079d281ce1bf781ac243d3580"}
Sep 16 02:41:18.234137 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:18.234089    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w" podStartSLOduration=1.234075185 podStartE2EDuration="1.234075185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:41:18.232059124 +0000 UTC m=+1377.662120509" watchObservedRunningTime="2026-09-16 02:41:18.234075185 +0000 UTC m=+1377.664136571"
Sep 16 02:41:27.463964 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:27.463922    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:27.464374 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:27.463982    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:27.476399 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:27.476375    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:28.280422 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:28.280396    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:41:42.150122 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.150085    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"]
Sep 16 02:41:42.159849 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.157711    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.161221 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.161192    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73bfc1e9\"/\"default-dockercfg-5mdhb\""
Sep 16 02:41:42.161361 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.161192    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73bfc1e9\"/\"e2e-lifecycle-model-73bfc1e9-kserve-self-signed-certs\""
Sep 16 02:41:42.161464 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.161203    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73bfc1e9\"/\"openshift-service-ca.crt\""
Sep 16 02:41:42.161570 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.161552    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73bfc1e9\"/\"kube-root-ca.crt\""
Sep 16 02:41:42.162330 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.162308    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"]
Sep 16 02:41:42.310787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.310756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-home\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.310979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.310809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-dshm\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.310979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.310856    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tmp-dir\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.310979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.310886    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-model-cache\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.310979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.310949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tls-certs\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.311145 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.310980    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5v54\" (UniqueName: \"kubernetes.io/projected/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-kube-api-access-z5v54\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.411930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.411843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-model-cache\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.411930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.411903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tls-certs\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.412157 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.411937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5v54\" (UniqueName: \"kubernetes.io/projected/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-kube-api-access-z5v54\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.412157 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.411985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-home\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.412157 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.412030    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-dshm\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.412157 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.412082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tmp-dir\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.412385 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.412267    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-model-cache\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.412385 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.412364    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-home\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.412540 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.412526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tmp-dir\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.414269 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.414246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-dshm\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.414584 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.414560    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tls-certs\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.418774 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.418754    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5v54\" (UniqueName: \"kubernetes.io/projected/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-kube-api-access-z5v54\") pod \"e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") " pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.469906 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.469883    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:42.745373 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:42.745351    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"]
Sep 16 02:41:42.747027 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:41:42.746998    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4efe407_8e5c_46d4_8fec_d4d25a9f3f4a.slice/crio-83e5237017610374958cfa7034f795f09e54b43a368e2d97241b55b4a427461e WatchSource:0}: Error finding container 83e5237017610374958cfa7034f795f09e54b43a368e2d97241b55b4a427461e: Status 404 returned error can't find the container with id 83e5237017610374958cfa7034f795f09e54b43a368e2d97241b55b4a427461e
Sep 16 02:41:43.347273 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:43.347239    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk" event={"ID":"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a","Type":"ContainerStarted","Data":"5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443"}
Sep 16 02:41:43.347693 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:43.347281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk" event={"ID":"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a","Type":"ContainerStarted","Data":"83e5237017610374958cfa7034f795f09e54b43a368e2d97241b55b4a427461e"}
Sep 16 02:41:43.363512 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:43.363469    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk" podStartSLOduration=1.363452692 podStartE2EDuration="1.363452692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:41:43.362529314 +0000 UTC m=+1402.792590703" watchObservedRunningTime="2026-09-16 02:41:43.363452692 +0000 UTC m=+1402.793514074"
Sep 16 02:41:52.470733 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:52.470701    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:52.471127 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:52.470753    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:52.483405 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:52.483376    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:41:53.113868 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:53.113837    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:41:53.414416 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:41:53.414326    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:42:02.731225 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:02.731187    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"]
Sep 16 02:42:17.750937 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:17.750857    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w" podUID="88e85920-c29c-4100-a5ab-c0d6f7c34256" containerName="main" containerID="cri-o://c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6" gracePeriod=45
Sep 16 02:42:17.992925 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:17.992904    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:42:18.037614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037539    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-home\") pod \"88e85920-c29c-4100-a5ab-c0d6f7c34256\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") "
Sep 16 02:42:18.037614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037575    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-tmp-dir\") pod \"88e85920-c29c-4100-a5ab-c0d6f7c34256\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") "
Sep 16 02:42:18.037835 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037634    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-model-cache\") pod \"88e85920-c29c-4100-a5ab-c0d6f7c34256\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") "
Sep 16 02:42:18.037835 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037675    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e85920-c29c-4100-a5ab-c0d6f7c34256-tls-certs\") pod \"88e85920-c29c-4100-a5ab-c0d6f7c34256\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") "
Sep 16 02:42:18.037835 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037732    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m87v9\" (UniqueName: \"kubernetes.io/projected/88e85920-c29c-4100-a5ab-c0d6f7c34256-kube-api-access-m87v9\") pod \"88e85920-c29c-4100-a5ab-c0d6f7c34256\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") "
Sep 16 02:42:18.037835 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037787    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-dshm\") pod \"88e85920-c29c-4100-a5ab-c0d6f7c34256\" (UID: \"88e85920-c29c-4100-a5ab-c0d6f7c34256\") "
Sep 16 02:42:18.037835 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037811    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-home" (OuterVolumeSpecName: "home") pod "88e85920-c29c-4100-a5ab-c0d6f7c34256" (UID: "88e85920-c29c-4100-a5ab-c0d6f7c34256"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:18.038113 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037849    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88e85920-c29c-4100-a5ab-c0d6f7c34256" (UID: "88e85920-c29c-4100-a5ab-c0d6f7c34256"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:18.038113 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.037884    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-model-cache" (OuterVolumeSpecName: "model-cache") pod "88e85920-c29c-4100-a5ab-c0d6f7c34256" (UID: "88e85920-c29c-4100-a5ab-c0d6f7c34256"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:18.038229 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.038132    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:18.038229 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.038146    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:18.038229 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.038155    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:18.039773 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.039750    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-dshm" (OuterVolumeSpecName: "dshm") pod "88e85920-c29c-4100-a5ab-c0d6f7c34256" (UID: "88e85920-c29c-4100-a5ab-c0d6f7c34256"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:18.039933 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.039910    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e85920-c29c-4100-a5ab-c0d6f7c34256-kube-api-access-m87v9" (OuterVolumeSpecName: "kube-api-access-m87v9") pod "88e85920-c29c-4100-a5ab-c0d6f7c34256" (UID: "88e85920-c29c-4100-a5ab-c0d6f7c34256"). InnerVolumeSpecName "kube-api-access-m87v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:42:18.039992 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.039916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e85920-c29c-4100-a5ab-c0d6f7c34256-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88e85920-c29c-4100-a5ab-c0d6f7c34256" (UID: "88e85920-c29c-4100-a5ab-c0d6f7c34256"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:42:18.138647 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.138611    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e85920-c29c-4100-a5ab-c0d6f7c34256-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:18.138647 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.138641    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m87v9\" (UniqueName: \"kubernetes.io/projected/88e85920-c29c-4100-a5ab-c0d6f7c34256-kube-api-access-m87v9\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:18.138647 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.138650    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e85920-c29c-4100-a5ab-c0d6f7c34256-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:18.530470 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.530423    2556 generic.go:356] "Generic (PLEG): container finished" podID="88e85920-c29c-4100-a5ab-c0d6f7c34256" containerID="c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6" exitCode=0
Sep 16 02:42:18.530679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.530476    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"
Sep 16 02:42:18.530679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.530531    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w" event={"ID":"88e85920-c29c-4100-a5ab-c0d6f7c34256","Type":"ContainerDied","Data":"c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6"}
Sep 16 02:42:18.530679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.530617    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w" event={"ID":"88e85920-c29c-4100-a5ab-c0d6f7c34256","Type":"ContainerDied","Data":"547c6315251ed37c9d3b26694ec4c3b08662557079d281ce1bf781ac243d3580"}
Sep 16 02:42:18.530679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.530648    2556 scope.go:117] "RemoveContainer" containerID="c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6"
Sep 16 02:42:18.541249 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.541233    2556 scope.go:117] "RemoveContainer" containerID="c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6"
Sep 16 02:42:18.541623 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:42:18.541602    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6\": container with ID starting with c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6 not found: ID does not exist" containerID="c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6"
Sep 16 02:42:18.541725 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.541631    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6"} err="failed to get container status \"c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6\": rpc error: code = NotFound desc = could not find container \"c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6\": container with ID starting with c6ce7f6e887aa1c133fe57e9cfd33a27f8283e3632994e6a1b2e8801e64489e6 not found: ID does not exist"
Sep 16 02:42:18.551953 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.551926    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"]
Sep 16 02:42:18.553792 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:18.553772    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-eee708/sub-test-model-eee708-kserve-fccf4c6dc-xcz5w"]
Sep 16 02:42:19.116073 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:19.116038    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e85920-c29c-4100-a5ab-c0d6f7c34256" path="/var/lib/kubelet/pods/88e85920-c29c-4100-a5ab-c0d6f7c34256/volumes"
Sep 16 02:42:26.465943 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.465909    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"]
Sep 16 02:42:26.466585 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.466151    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p" podUID="01b72374-4bbd-48be-a9b4-417a17240a6b" containerName="maas-api" containerID="cri-o://fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55" gracePeriod=30
Sep 16 02:42:26.722098 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.722038    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:42:26.816600 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.816564    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-maas-api-tls\") pod \"01b72374-4bbd-48be-a9b4-417a17240a6b\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") "
Sep 16 02:42:26.816784 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.816629    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpvd8\" (UniqueName: \"kubernetes.io/projected/01b72374-4bbd-48be-a9b4-417a17240a6b-kube-api-access-fpvd8\") pod \"01b72374-4bbd-48be-a9b4-417a17240a6b\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") "
Sep 16 02:42:26.816784 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.816697    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72374-4bbd-48be-a9b4-417a17240a6b-openshift-service-ca\") pod \"01b72374-4bbd-48be-a9b4-417a17240a6b\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") "
Sep 16 02:42:26.816784 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.816737    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-metrics-certs\") pod \"01b72374-4bbd-48be-a9b4-417a17240a6b\" (UID: \"01b72374-4bbd-48be-a9b4-417a17240a6b\") "
Sep 16 02:42:26.817137 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.817105    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b72374-4bbd-48be-a9b4-417a17240a6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01b72374-4bbd-48be-a9b4-417a17240a6b" (UID: "01b72374-4bbd-48be-a9b4-417a17240a6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:42:26.818813 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.818783    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b72374-4bbd-48be-a9b4-417a17240a6b-kube-api-access-fpvd8" (OuterVolumeSpecName: "kube-api-access-fpvd8") pod "01b72374-4bbd-48be-a9b4-417a17240a6b" (UID: "01b72374-4bbd-48be-a9b4-417a17240a6b"). InnerVolumeSpecName "kube-api-access-fpvd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:42:26.818928 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.818879    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01b72374-4bbd-48be-a9b4-417a17240a6b" (UID: "01b72374-4bbd-48be-a9b4-417a17240a6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:42:26.819184 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.819166    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "01b72374-4bbd-48be-a9b4-417a17240a6b" (UID: "01b72374-4bbd-48be-a9b4-417a17240a6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:42:26.917964 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.917937    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:26.917964 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.917962    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvd8\" (UniqueName: \"kubernetes.io/projected/01b72374-4bbd-48be-a9b4-417a17240a6b-kube-api-access-fpvd8\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:26.917964 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.917973    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b72374-4bbd-48be-a9b4-417a17240a6b-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:26.918169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:26.917981    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b72374-4bbd-48be-a9b4-417a17240a6b-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:27.577139 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.577110    2556 generic.go:356] "Generic (PLEG): container finished" podID="01b72374-4bbd-48be-a9b4-417a17240a6b" containerID="fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55" exitCode=0
Sep 16 02:42:27.577587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.577159    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"
Sep 16 02:42:27.577587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.577206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p" event={"ID":"01b72374-4bbd-48be-a9b4-417a17240a6b","Type":"ContainerDied","Data":"fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55"}
Sep 16 02:42:27.577587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.577256    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p" event={"ID":"01b72374-4bbd-48be-a9b4-417a17240a6b","Type":"ContainerDied","Data":"1cc9b3685c4c3e0ae34dbd9be3fd1bf7d7576b756bf146b90bd7fea241037c43"}
Sep 16 02:42:27.577587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.577280    2556 scope.go:117] "RemoveContainer" containerID="fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55"
Sep 16 02:42:27.586200 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.586185    2556 scope.go:117] "RemoveContainer" containerID="fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55"
Sep 16 02:42:27.586478 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:42:27.586456    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55\": container with ID starting with fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55 not found: ID does not exist" containerID="fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55"
Sep 16 02:42:27.586524 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.586488    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55"} err="failed to get container status \"fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55\": rpc error: code = NotFound desc = could not find container \"fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55\": container with ID starting with fcf86834f798ff982867e63a41bdbc231b1272e94f6075784d0aac3b9cb9da55 not found: ID does not exist"
Sep 16 02:42:27.594954 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.594932    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"]
Sep 16 02:42:27.597981 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:27.597958    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-73bfc1e9-6786dcd499-fv94p"]
Sep 16 02:42:29.114584 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:29.114553    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b72374-4bbd-48be-a9b4-417a17240a6b" path="/var/lib/kubelet/pods/01b72374-4bbd-48be-a9b4-417a17240a6b/volumes"
Sep 16 02:42:34.813326 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:34.813288    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"]
Sep 16 02:42:49.832412 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:49.832347    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk" podUID="e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" containerName="main" containerID="cri-o://5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443" gracePeriod=45
Sep 16 02:42:50.085609 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.085546    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:42:50.228965 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.228930    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5v54\" (UniqueName: \"kubernetes.io/projected/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-kube-api-access-z5v54\") pod \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") "
Sep 16 02:42:50.229132 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.228984    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-home\") pod \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") "
Sep 16 02:42:50.229132 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229028    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tls-certs\") pod \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") "
Sep 16 02:42:50.229132 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229070    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-model-cache\") pod \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") "
Sep 16 02:42:50.229132 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229095    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tmp-dir\") pod \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") "
Sep 16 02:42:50.229516 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229158    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-dshm\") pod \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\" (UID: \"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a\") "
Sep 16 02:42:50.229516 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229344    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-home" (OuterVolumeSpecName: "home") pod "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" (UID: "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:50.229516 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229339    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" (UID: "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:50.229516 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229359    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" (UID: "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:50.229716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229556    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:50.229716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229584    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:50.229716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.229599    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:50.231212 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.231188    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-dshm" (OuterVolumeSpecName: "dshm") pod "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" (UID: "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:42:50.231297 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.231272    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-kube-api-access-z5v54" (OuterVolumeSpecName: "kube-api-access-z5v54") pod "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" (UID: "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a"). InnerVolumeSpecName "kube-api-access-z5v54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:42:50.231350 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.231325    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" (UID: "e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:42:50.330850 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.330804    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5v54\" (UniqueName: \"kubernetes.io/projected/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-kube-api-access-z5v54\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:50.330850 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.330844    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:50.330850 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.330858    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:42:50.698852 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.698814    2556 generic.go:356] "Generic (PLEG): container finished" podID="e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" containerID="5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443" exitCode=0
Sep 16 02:42:50.699060 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.698889    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"
Sep 16 02:42:50.699060 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.698907    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk" event={"ID":"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a","Type":"ContainerDied","Data":"5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443"}
Sep 16 02:42:50.699060 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.698950    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk" event={"ID":"e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a","Type":"ContainerDied","Data":"83e5237017610374958cfa7034f795f09e54b43a368e2d97241b55b4a427461e"}
Sep 16 02:42:50.699060 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.698971    2556 scope.go:117] "RemoveContainer" containerID="5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443"
Sep 16 02:42:50.708488 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.708469    2556 scope.go:117] "RemoveContainer" containerID="5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443"
Sep 16 02:42:50.708771 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:42:50.708752    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443\": container with ID starting with 5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443 not found: ID does not exist" containerID="5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443"
Sep 16 02:42:50.708839 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.708778    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443"} err="failed to get container status \"5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443\": rpc error: code = NotFound desc = could not find container \"5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443\": container with ID starting with 5cfda6d540acee41ff3c324d7d022516bdcd7f44815918edc2946f5888f53443 not found: ID does not exist"
Sep 16 02:42:50.720392 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.720365    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"]
Sep 16 02:42:50.723657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:50.723635    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-73bfc1e9/e2e-lifecycle-model-73bfc1e9-kserve-68fc8dccf5-27snk"]
Sep 16 02:42:51.115709 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:42:51.115627    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" path="/var/lib/kubelet/pods/e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a/volumes"
Sep 16 02:43:13.107732 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:13.107689    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:43:21.060365 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:21.060333    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:43:21.066625 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:21.066600    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:43:30.008876 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.008793    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"]
Sep 16 02:43:30.010015 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.009988    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b72374-4bbd-48be-a9b4-417a17240a6b" containerName="maas-api"
Sep 16 02:43:30.010151 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010026    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b72374-4bbd-48be-a9b4-417a17240a6b" containerName="maas-api"
Sep 16 02:43:30.010151 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010067    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" containerName="main"
Sep 16 02:43:30.010151 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010076    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" containerName="main"
Sep 16 02:43:30.010151 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010106    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e85920-c29c-4100-a5ab-c0d6f7c34256" containerName="main"
Sep 16 02:43:30.010151 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010114    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e85920-c29c-4100-a5ab-c0d6f7c34256" containerName="main"
Sep 16 02:43:30.010504 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010220    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4efe407-8e5c-46d4-8fec-d4d25a9f3f4a" containerName="main"
Sep 16 02:43:30.010504 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010230    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e85920-c29c-4100-a5ab-c0d6f7c34256" containerName="main"
Sep 16 02:43:30.010504 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.010243    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b72374-4bbd-48be-a9b4-417a17240a6b" containerName="maas-api"
Sep 16 02:43:30.015196 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.014949    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.017276 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.017252    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-10a6b9-serving-cert\""
Sep 16 02:43:30.024565 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.024537    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"]
Sep 16 02:43:30.196716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.196675    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bc698c-553e-43f4-a546-6547924a5d8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.196940 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.196873    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-metrics-certs\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.196940 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.196932    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.197088 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.197061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lnq\" (UniqueName: \"kubernetes.io/projected/b6bc698c-553e-43f4-a546-6547924a5d8b-kube-api-access-d9lnq\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.298029 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.297931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lnq\" (UniqueName: \"kubernetes.io/projected/b6bc698c-553e-43f4-a546-6547924a5d8b-kube-api-access-d9lnq\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.298029 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.297996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bc698c-553e-43f4-a546-6547924a5d8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.298287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.298123    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-metrics-certs\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.298287 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.298153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.298287 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:43:30.298281    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-serving-cert: secret "maas-api-e2e-ipp-a-10a6b9-serving-cert" not found
Sep 16 02:43:30.298470 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:43:30.298391    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls podName:b6bc698c-553e-43f4-a546-6547924a5d8b nodeName:}" failed. No retries permitted until 2026-09-16 02:43:30.79834126 +0000 UTC m=+1510.228402639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls") pod "maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8" (UID: "b6bc698c-553e-43f4-a546-6547924a5d8b") : secret "maas-api-e2e-ipp-a-10a6b9-serving-cert" not found
Sep 16 02:43:30.298736 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.298711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bc698c-553e-43f4-a546-6547924a5d8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.300601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.300579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-metrics-certs\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.314401 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.314372    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lnq\" (UniqueName: \"kubernetes.io/projected/b6bc698c-553e-43f4-a546-6547924a5d8b-kube-api-access-d9lnq\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.804502 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.804456    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.807194 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.807165    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:30.929693 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:30.929657    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:31.205313 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:31.201633    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"]
Sep 16 02:43:31.922735 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:31.922697    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8" event={"ID":"b6bc698c-553e-43f4-a546-6547924a5d8b","Type":"ContainerStarted","Data":"b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776"}
Sep 16 02:43:31.922905 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:31.922759    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8" event={"ID":"b6bc698c-553e-43f4-a546-6547924a5d8b","Type":"ContainerStarted","Data":"6f84012ab1a389a48af5284d390cb16f1cc04b2f383fe7b95022ee2d25d55ff0"}
Sep 16 02:43:31.922905 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:31.922827    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:31.943535 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:31.943484    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8" podStartSLOduration=2.609398871 podStartE2EDuration="2.943465489s" totalImagesPullingTime="334.066618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:43:31.942567123 +0000 UTC m=+1511.372628511" watchObservedRunningTime="2026-09-16 02:43:31.943465489 +0000 UTC m=+1511.373526878"
Sep 16 02:43:37.933357 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:37.933319    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:43:59.796512 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.796449    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"]
Sep 16 02:43:59.800380 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.800351    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.802571 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.802542    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4fe974-serving-cert\""
Sep 16 02:43:59.808793 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.808761    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"]
Sep 16 02:43:59.867459 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.867399    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hndz\" (UniqueName: \"kubernetes.io/projected/8bec56eb-142b-4d14-8b0e-02fc993d116a-kube-api-access-8hndz\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.867622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.867540    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec56eb-142b-4d14-8b0e-02fc993d116a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.867622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.867582    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-metrics-certs\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.867713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.867622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.968588 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.968552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hndz\" (UniqueName: \"kubernetes.io/projected/8bec56eb-142b-4d14-8b0e-02fc993d116a-kube-api-access-8hndz\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.968777 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.968622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec56eb-142b-4d14-8b0e-02fc993d116a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.968777 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.968646    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-metrics-certs\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.968777 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.968678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.968960 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:43:59.968780    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-serving-cert: secret "maas-api-e2e-ipp-b-4fe974-serving-cert" not found
Sep 16 02:43:59.968960 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:43:59.968848    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls podName:8bec56eb-142b-4d14-8b0e-02fc993d116a nodeName:}" failed. No retries permitted until 2026-09-16 02:44:00.468831728 +0000 UTC m=+1539.898893094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls") pod "maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" (UID: "8bec56eb-142b-4d14-8b0e-02fc993d116a") : secret "maas-api-e2e-ipp-b-4fe974-serving-cert" not found
Sep 16 02:43:59.969379 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.969356    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec56eb-142b-4d14-8b0e-02fc993d116a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.971042 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.971023    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-metrics-certs\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:43:59.976215 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:43:59.976192    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hndz\" (UniqueName: \"kubernetes.io/projected/8bec56eb-142b-4d14-8b0e-02fc993d116a-kube-api-access-8hndz\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:44:00.472812 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:00.472772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:44:00.472996 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:44:00.472893    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-serving-cert: secret "maas-api-e2e-ipp-b-4fe974-serving-cert" not found
Sep 16 02:44:00.472996 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:44:00.472946    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls podName:8bec56eb-142b-4d14-8b0e-02fc993d116a nodeName:}" failed. No retries permitted until 2026-09-16 02:44:01.47293264 +0000 UTC m=+1540.902994003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls") pod "maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" (UID: "8bec56eb-142b-4d14-8b0e-02fc993d116a") : secret "maas-api-e2e-ipp-b-4fe974-serving-cert" not found
Sep 16 02:44:01.483512 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:01.483478    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:44:01.485918 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:01.485893    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:44:01.615327 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:01.615294    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:44:01.885854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:01.885826    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"]
Sep 16 02:44:01.887685 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:44:01.887652    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bec56eb_142b_4d14_8b0e_02fc993d116a.slice/crio-1a45da4c5d4357460338eba84e72ea783c9310454582fc0d51424ad6c4469fab WatchSource:0}: Error finding container 1a45da4c5d4357460338eba84e72ea783c9310454582fc0d51424ad6c4469fab: Status 404 returned error can't find the container with id 1a45da4c5d4357460338eba84e72ea783c9310454582fc0d51424ad6c4469fab
Sep 16 02:44:02.080739 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:02.080702    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" event={"ID":"8bec56eb-142b-4d14-8b0e-02fc993d116a","Type":"ContainerStarted","Data":"1a45da4c5d4357460338eba84e72ea783c9310454582fc0d51424ad6c4469fab"}
Sep 16 02:44:03.087493 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:03.087453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" event={"ID":"8bec56eb-142b-4d14-8b0e-02fc993d116a","Type":"ContainerStarted","Data":"2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f"}
Sep 16 02:44:03.087855 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:03.087579    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:44:03.104890 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:03.104844    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" podStartSLOduration=3.682240981 podStartE2EDuration="4.104830214s" totalImagesPullingTime="422.589233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:03.102161633 +0000 UTC m=+1542.532223021" watchObservedRunningTime="2026-09-16 02:44:03.104830214 +0000 UTC m=+1542.534891638"
Sep 16 02:44:09.098990 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:09.098957    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:44:23.108208 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:23.108163    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:44:26.400618 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:26.400587    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-p6mx9_c9582f82-fc4a-448c-99e4-4fc2fc2d118d/payload-processing/0.log"
Sep 16 02:44:26.508380 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:26.508321    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-p6mx9_c9582f82-fc4a-448c-99e4-4fc2fc2d118d/payload-processing/0.log"
Sep 16 02:44:37.719816 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.719770    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"]
Sep 16 02:44:37.725218 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.725192    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.727423 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.727397    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4178a6-serving-cert\""
Sep 16 02:44:37.747031 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.743046    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"]
Sep 16 02:44:37.819292 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.819249    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdpj\" (UniqueName: \"kubernetes.io/projected/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-kube-api-access-qrdpj\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.819488 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.819333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-metrics-certs\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.819488 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.819363    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-openshift-service-ca\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.819488 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.819386    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.920903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.920859    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-openshift-service-ca\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.921071 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.920926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.921071 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.921006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdpj\" (UniqueName: \"kubernetes.io/projected/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-kube-api-access-qrdpj\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.921169 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:44:37.921075    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-serving-cert: secret "maas-api-e2e-api-a-4178a6-serving-cert" not found
Sep 16 02:44:37.921169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.921084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-metrics-certs\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.921268 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:44:37.921165    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls podName:d8fac89a-bb94-4cc6-91df-8d22b5513f3d nodeName:}" failed. No retries permitted until 2026-09-16 02:44:38.421144084 +0000 UTC m=+1577.851205461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls") pod "maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw" (UID: "d8fac89a-bb94-4cc6-91df-8d22b5513f3d") : secret "maas-api-e2e-api-a-4178a6-serving-cert" not found
Sep 16 02:44:37.921566 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.921545    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-openshift-service-ca\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.923546 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.923521    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-metrics-certs\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:37.928600 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:37.928579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdpj\" (UniqueName: \"kubernetes.io/projected/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-kube-api-access-qrdpj\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:38.426255 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:38.426205    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:38.428707 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:38.428672    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls\") pod \"maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:38.647583 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:38.647545    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:38.919174 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:38.919146    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"]
Sep 16 02:44:38.920972 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:44:38.920943    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fac89a_bb94_4cc6_91df_8d22b5513f3d.slice/crio-f57a4448b2d6d5a1e37d1a37c9a715a762bd4dd42780e1751330704c7e510ad7 WatchSource:0}: Error finding container f57a4448b2d6d5a1e37d1a37c9a715a762bd4dd42780e1751330704c7e510ad7: Status 404 returned error can't find the container with id f57a4448b2d6d5a1e37d1a37c9a715a762bd4dd42780e1751330704c7e510ad7
Sep 16 02:44:39.271041 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:39.271004    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw" event={"ID":"d8fac89a-bb94-4cc6-91df-8d22b5513f3d","Type":"ContainerStarted","Data":"f57a4448b2d6d5a1e37d1a37c9a715a762bd4dd42780e1751330704c7e510ad7"}
Sep 16 02:44:40.278567 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:40.278526    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw" event={"ID":"d8fac89a-bb94-4cc6-91df-8d22b5513f3d","Type":"ContainerStarted","Data":"e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97"}
Sep 16 02:44:40.278972 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:40.278596    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:44:40.294466 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:40.294397    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw" podStartSLOduration=2.766070026 podStartE2EDuration="3.294383519s" totalImagesPullingTime="528.313493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:44:40.29277987 +0000 UTC m=+1579.722841255" watchObservedRunningTime="2026-09-16 02:44:40.294383519 +0000 UTC m=+1579.724444905"
Sep 16 02:44:46.289275 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:44:46.289243    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:45:03.753622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.753586    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"]
Sep 16 02:45:03.758089 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.758067    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.760376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.760352    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-10a6b9\"/\"default-dockercfg-kvsnm\""
Sep 16 02:45:03.761646 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.761624    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-10a6b9\"/\"ipp-route-10a6b9-kserve-self-signed-certs\""
Sep 16 02:45:03.761931 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.761662    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-10a6b9\"/\"openshift-service-ca.crt\""
Sep 16 02:45:03.761931 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.761892    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-10a6b9\"/\"kube-root-ca.crt\""
Sep 16 02:45:03.764821 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.764800    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"]
Sep 16 02:45:03.873566 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.873536    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e705720-7a42-4466-8f3a-6ff07a042f0f-tls-certs\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.873566 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.873570    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-model-cache\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.873829 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.873602    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-dshm\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.873829 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.873704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-home\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.873930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.873845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-tmp-dir\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.873930 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.873880    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69kk\" (UniqueName: \"kubernetes.io/projected/6e705720-7a42-4466-8f3a-6ff07a042f0f-kube-api-access-k69kk\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975258 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975223    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e705720-7a42-4466-8f3a-6ff07a042f0f-tls-certs\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975258 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-model-cache\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975541 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975281    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-dshm\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975541 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975306    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-home\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975541 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-tmp-dir\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975541 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69kk\" (UniqueName: \"kubernetes.io/projected/6e705720-7a42-4466-8f3a-6ff07a042f0f-kube-api-access-k69kk\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975758 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-model-cache\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975758 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-home\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.975825 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.975778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-tmp-dir\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.977610 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.977589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-dshm\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.977723 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.977707    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e705720-7a42-4466-8f3a-6ff07a042f0f-tls-certs\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:03.982699 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:03.982679    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69kk\" (UniqueName: \"kubernetes.io/projected/6e705720-7a42-4466-8f3a-6ff07a042f0f-kube-api-access-k69kk\") pod \"ipp-route-10a6b9-kserve-56c99699f9-rzmqk\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") " pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:04.070851 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:04.070774    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:04.346776 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:04.346592    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"]
Sep 16 02:45:04.349532 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:45:04.349507    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e705720_7a42_4466_8f3a_6ff07a042f0f.slice/crio-be9ea64675bd46693d44e58f5579ee166c4e716853a271ff396b2c02ce695403 WatchSource:0}: Error finding container be9ea64675bd46693d44e58f5579ee166c4e716853a271ff396b2c02ce695403: Status 404 returned error can't find the container with id be9ea64675bd46693d44e58f5579ee166c4e716853a271ff396b2c02ce695403
Sep 16 02:45:04.408849 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:04.408783    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk" event={"ID":"6e705720-7a42-4466-8f3a-6ff07a042f0f","Type":"ContainerStarted","Data":"be9ea64675bd46693d44e58f5579ee166c4e716853a271ff396b2c02ce695403"}
Sep 16 02:45:05.415980 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:05.415945    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk" event={"ID":"6e705720-7a42-4466-8f3a-6ff07a042f0f","Type":"ContainerStarted","Data":"c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603"}
Sep 16 02:45:05.432343 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:05.432297    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk" podStartSLOduration=2.432281461 podStartE2EDuration="2.432281461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:05.429699039 +0000 UTC m=+1604.859760425" watchObservedRunningTime="2026-09-16 02:45:05.432281461 +0000 UTC m=+1604.862342876"
Sep 16 02:45:12.006832 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.006799    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"]
Sep 16 02:45:12.010679 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.010657    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.012882 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.012855    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1a50f4-serving-cert\""
Sep 16 02:45:12.021291 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.021186    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"]
Sep 16 02:45:12.045277 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.045232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-metrics-certs\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.045446 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.045292    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwc8\" (UniqueName: \"kubernetes.io/projected/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-kube-api-access-sbwc8\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.045446 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.045329    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.045446 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.045372    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-maas-api-tls\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.146220 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.146182    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.146391 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.146250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-maas-api-tls\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.146391 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.146364    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-metrics-certs\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.146496 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.146399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwc8\" (UniqueName: \"kubernetes.io/projected/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-kube-api-access-sbwc8\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.146984 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.146960    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.148798 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.148767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-metrics-certs\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.148918 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.148856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-maas-api-tls\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.153519 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.153496    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwc8\" (UniqueName: \"kubernetes.io/projected/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-kube-api-access-sbwc8\") pod \"maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.324734 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.324647    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:12.595177 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:12.595133    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"]
Sep 16 02:45:13.458226 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:13.458193    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs" event={"ID":"ef480d1a-af1a-41f5-b9a9-e128e3fd401f","Type":"ContainerStarted","Data":"af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a"}
Sep 16 02:45:13.458682 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:13.458245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs" event={"ID":"ef480d1a-af1a-41f5-b9a9-e128e3fd401f","Type":"ContainerStarted","Data":"c7ad2dc58b9f009f905507c59e662edc0c81690bd1e7d78bd42fe5bcb2e40f63"}
Sep 16 02:45:13.458682 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:13.458284    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:13.473722 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:13.473668    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs" podStartSLOduration=2.032770749 podStartE2EDuration="2.473648122s" totalImagesPullingTime="440.877373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:13.473109633 +0000 UTC m=+1612.903171022" watchObservedRunningTime="2026-09-16 02:45:13.473648122 +0000 UTC m=+1612.903709510"
Sep 16 02:45:14.071808 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:14.071772    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:14.071987 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:14.071828    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:14.084966 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:14.084932    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:14.475842 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:14.475813    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:45:19.469455 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:19.469408    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:45:34.107284 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:34.107248    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:45:42.641176 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:42.641150    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-p6mx9_c9582f82-fc4a-448c-99e4-4fc2fc2d118d/payload-processing/0.log"
Sep 16 02:45:42.749379 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:42.749347    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-p6mx9_c9582f82-fc4a-448c-99e4-4fc2fc2d118d/payload-processing/0.log"
Sep 16 02:45:52.139787 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.139751    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"]
Sep 16 02:45:52.143046 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.143027    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.145043 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.145023    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-342933-serving-cert\""
Sep 16 02:45:52.151976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.151953    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"]
Sep 16 02:45:52.228717 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.228678    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p96g\" (UniqueName: \"kubernetes.io/projected/7dd93203-0537-4f70-aa67-b972547f824d-kube-api-access-8p96g\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.228884 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.228727    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.228884 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.228865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd93203-0537-4f70-aa67-b972547f824d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.229003 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.228895    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.329406 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.329373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd93203-0537-4f70-aa67-b972547f824d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.329601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.329417    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.329601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.329484    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p96g\" (UniqueName: \"kubernetes.io/projected/7dd93203-0537-4f70-aa67-b972547f824d-kube-api-access-8p96g\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.329601 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.329526    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.329783 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:45:52.329643    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-serving-cert: secret "maas-api-e2e-ipp-cleanup-342933-serving-cert" not found
Sep 16 02:45:52.329783 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:45:52.329719    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls podName:7dd93203-0537-4f70-aa67-b972547f824d nodeName:}" failed. No retries permitted until 2026-09-16 02:45:52.829697998 +0000 UTC m=+1652.259759379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6" (UID: "7dd93203-0537-4f70-aa67-b972547f824d") : secret "maas-api-e2e-ipp-cleanup-342933-serving-cert" not found
Sep 16 02:45:52.329995 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.329973    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd93203-0537-4f70-aa67-b972547f824d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.331819 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.331801    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.337272 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.337244    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p96g\" (UniqueName: \"kubernetes.io/projected/7dd93203-0537-4f70-aa67-b972547f824d-kube-api-access-8p96g\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.834138 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.834101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:52.836447 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:52.836410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:53.055614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:53.055570    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:53.327191 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:53.327167    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"]
Sep 16 02:45:53.329223 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:45:53.329195    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd93203_0537_4f70_aa67_b972547f824d.slice/crio-eb9570e702a24708fd6bc6d3cf4b8739a73c9fa846aa55be7f0ca85e9d9e7a7e WatchSource:0}: Error finding container eb9570e702a24708fd6bc6d3cf4b8739a73c9fa846aa55be7f0ca85e9d9e7a7e: Status 404 returned error can't find the container with id eb9570e702a24708fd6bc6d3cf4b8739a73c9fa846aa55be7f0ca85e9d9e7a7e
Sep 16 02:45:53.330627 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:53.330609    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 02:45:53.686696 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:53.686664    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6" event={"ID":"7dd93203-0537-4f70-aa67-b972547f824d","Type":"ContainerStarted","Data":"eb9570e702a24708fd6bc6d3cf4b8739a73c9fa846aa55be7f0ca85e9d9e7a7e"}
Sep 16 02:45:54.692956 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:54.692918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6" event={"ID":"7dd93203-0537-4f70-aa67-b972547f824d","Type":"ContainerStarted","Data":"93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651"}
Sep 16 02:45:54.693382 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:54.693001    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:45:54.708598 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:54.708555    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6" podStartSLOduration=2.409310528 podStartE2EDuration="2.708542186s" totalImagesPullingTime="299.231658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:45:54.707402867 +0000 UTC m=+1654.137464254" watchObservedRunningTime="2026-09-16 02:45:54.708542186 +0000 UTC m=+1654.138603573"
Sep 16 02:45:58.107639 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.107601    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"]
Sep 16 02:45:58.108148 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.107882    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw" podUID="d8fac89a-bb94-4cc6-91df-8d22b5513f3d" containerName="maas-api" containerID="cri-o://e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97" gracePeriod=30
Sep 16 02:45:58.373291 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.373226    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:45:58.487241 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.487205    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-metrics-certs\") pod \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") "
Sep 16 02:45:58.487409 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.487265    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrdpj\" (UniqueName: \"kubernetes.io/projected/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-kube-api-access-qrdpj\") pod \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") "
Sep 16 02:45:58.487409 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.487299    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-openshift-service-ca\") pod \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") "
Sep 16 02:45:58.487409 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.487403    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls\") pod \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\" (UID: \"d8fac89a-bb94-4cc6-91df-8d22b5513f3d\") "
Sep 16 02:45:58.487763 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.487734    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8fac89a-bb94-4cc6-91df-8d22b5513f3d" (UID: "d8fac89a-bb94-4cc6-91df-8d22b5513f3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:45:58.489477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.489451    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d8fac89a-bb94-4cc6-91df-8d22b5513f3d" (UID: "d8fac89a-bb94-4cc6-91df-8d22b5513f3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:45:58.489477 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.489462    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-kube-api-access-qrdpj" (OuterVolumeSpecName: "kube-api-access-qrdpj") pod "d8fac89a-bb94-4cc6-91df-8d22b5513f3d" (UID: "d8fac89a-bb94-4cc6-91df-8d22b5513f3d"). InnerVolumeSpecName "kube-api-access-qrdpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:45:58.489667 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.489550    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8fac89a-bb94-4cc6-91df-8d22b5513f3d" (UID: "d8fac89a-bb94-4cc6-91df-8d22b5513f3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:45:58.588376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.588341    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdpj\" (UniqueName: \"kubernetes.io/projected/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-kube-api-access-qrdpj\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:45:58.588376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.588370    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:45:58.588376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.588379    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:45:58.588641 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.588388    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8fac89a-bb94-4cc6-91df-8d22b5513f3d-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:45:58.715556 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.715522    2556 generic.go:356] "Generic (PLEG): container finished" podID="d8fac89a-bb94-4cc6-91df-8d22b5513f3d" containerID="e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97" exitCode=0
Sep 16 02:45:58.715740 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.715570    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"
Sep 16 02:45:58.715740 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.715581    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw" event={"ID":"d8fac89a-bb94-4cc6-91df-8d22b5513f3d","Type":"ContainerDied","Data":"e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97"}
Sep 16 02:45:58.715740 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.715623    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw" event={"ID":"d8fac89a-bb94-4cc6-91df-8d22b5513f3d","Type":"ContainerDied","Data":"f57a4448b2d6d5a1e37d1a37c9a715a762bd4dd42780e1751330704c7e510ad7"}
Sep 16 02:45:58.715740 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.715644    2556 scope.go:117] "RemoveContainer" containerID="e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97"
Sep 16 02:45:58.725335 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.725315    2556 scope.go:117] "RemoveContainer" containerID="e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97"
Sep 16 02:45:58.725684 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:45:58.725663    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97\": container with ID starting with e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97 not found: ID does not exist" containerID="e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97"
Sep 16 02:45:58.725734 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.725693    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97"} err="failed to get container status \"e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97\": rpc error: code = NotFound desc = could not find container \"e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97\": container with ID starting with e6435ccec2b84f08604fa2f52a655ab8cf04c38569d5b2d9fc6024e53a74de97 not found: ID does not exist"
Sep 16 02:45:58.738553 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.738529    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"]
Sep 16 02:45:58.742262 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:58.742239    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4178a6-65ddd687c5-mrbkw"]
Sep 16 02:45:59.114486 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:45:59.114456    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8fac89a-bb94-4cc6-91df-8d22b5513f3d" path="/var/lib/kubelet/pods/d8fac89a-bb94-4cc6-91df-8d22b5513f3d/volumes"
Sep 16 02:46:00.704759 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:00.704730    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:46:18.255295 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.255260    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"]
Sep 16 02:46:18.255935 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.255492    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6" podUID="7dd93203-0537-4f70-aa67-b972547f824d" containerName="maas-api" containerID="cri-o://93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651" gracePeriod=30
Sep 16 02:46:18.516725 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.516658    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:46:18.591650 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.591611    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls\") pod \"7dd93203-0537-4f70-aa67-b972547f824d\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") "
Sep 16 02:46:18.591847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.591659    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p96g\" (UniqueName: \"kubernetes.io/projected/7dd93203-0537-4f70-aa67-b972547f824d-kube-api-access-8p96g\") pod \"7dd93203-0537-4f70-aa67-b972547f824d\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") "
Sep 16 02:46:18.591847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.591685    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-metrics-certs\") pod \"7dd93203-0537-4f70-aa67-b972547f824d\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") "
Sep 16 02:46:18.591847 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.591801    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd93203-0537-4f70-aa67-b972547f824d-openshift-service-ca\") pod \"7dd93203-0537-4f70-aa67-b972547f824d\" (UID: \"7dd93203-0537-4f70-aa67-b972547f824d\") "
Sep 16 02:46:18.592234 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.592208    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd93203-0537-4f70-aa67-b972547f824d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dd93203-0537-4f70-aa67-b972547f824d" (UID: "7dd93203-0537-4f70-aa67-b972547f824d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:18.594034 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.594002    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd93203-0537-4f70-aa67-b972547f824d-kube-api-access-8p96g" (OuterVolumeSpecName: "kube-api-access-8p96g") pod "7dd93203-0537-4f70-aa67-b972547f824d" (UID: "7dd93203-0537-4f70-aa67-b972547f824d"). InnerVolumeSpecName "kube-api-access-8p96g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:18.594487 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.594458    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dd93203-0537-4f70-aa67-b972547f824d" (UID: "7dd93203-0537-4f70-aa67-b972547f824d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:18.594603 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.594485    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7dd93203-0537-4f70-aa67-b972547f824d" (UID: "7dd93203-0537-4f70-aa67-b972547f824d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:18.693141 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.693104    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:18.693141 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.693136    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p96g\" (UniqueName: \"kubernetes.io/projected/7dd93203-0537-4f70-aa67-b972547f824d-kube-api-access-8p96g\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:18.693141 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.693147    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd93203-0537-4f70-aa67-b972547f824d-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:18.693381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.693155    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd93203-0537-4f70-aa67-b972547f824d-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:18.824986 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.824887    2556 generic.go:356] "Generic (PLEG): container finished" podID="7dd93203-0537-4f70-aa67-b972547f824d" containerID="93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651" exitCode=0
Sep 16 02:46:18.824986 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.824949    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"
Sep 16 02:46:18.825220 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.824983    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6" event={"ID":"7dd93203-0537-4f70-aa67-b972547f824d","Type":"ContainerDied","Data":"93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651"}
Sep 16 02:46:18.825220 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.825040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6" event={"ID":"7dd93203-0537-4f70-aa67-b972547f824d","Type":"ContainerDied","Data":"eb9570e702a24708fd6bc6d3cf4b8739a73c9fa846aa55be7f0ca85e9d9e7a7e"}
Sep 16 02:46:18.825220 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.825062    2556 scope.go:117] "RemoveContainer" containerID="93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651"
Sep 16 02:46:18.835960 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.835941    2556 scope.go:117] "RemoveContainer" containerID="93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651"
Sep 16 02:46:18.836292 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:46:18.836265    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651\": container with ID starting with 93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651 not found: ID does not exist" containerID="93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651"
Sep 16 02:46:18.836379 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.836307    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651"} err="failed to get container status \"93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651\": rpc error: code = NotFound desc = could not find container \"93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651\": container with ID starting with 93fcb52a475011415e5d09bcab592ff3d764d04db4008d3bb116ef2d407b9651 not found: ID does not exist"
Sep 16 02:46:18.851521 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.851488    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"]
Sep 16 02:46:18.854082 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:18.854051    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-342933-867dbfb55d-rghd6"]
Sep 16 02:46:19.115587 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:19.115511    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd93203-0537-4f70-aa67-b972547f824d" path="/var/lib/kubelet/pods/7dd93203-0537-4f70-aa67-b972547f824d/volumes"
Sep 16 02:46:23.032956 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.032919    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"]
Sep 16 02:46:23.033460 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.033184    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs" podUID="ef480d1a-af1a-41f5-b9a9-e128e3fd401f" containerName="maas-api" containerID="cri-o://af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a" gracePeriod=30
Sep 16 02:46:23.299628 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.299567    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:46:23.435716 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.435689    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-maas-api-tls\") pod \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") "
Sep 16 02:46:23.435980 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.435721    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-openshift-service-ca\") pod \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") "
Sep 16 02:46:23.435980 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.435780    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbwc8\" (UniqueName: \"kubernetes.io/projected/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-kube-api-access-sbwc8\") pod \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") "
Sep 16 02:46:23.435980 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.435831    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-metrics-certs\") pod \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\" (UID: \"ef480d1a-af1a-41f5-b9a9-e128e3fd401f\") "
Sep 16 02:46:23.436222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.436187    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef480d1a-af1a-41f5-b9a9-e128e3fd401f" (UID: "ef480d1a-af1a-41f5-b9a9-e128e3fd401f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:23.437866 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.437839    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef480d1a-af1a-41f5-b9a9-e128e3fd401f" (UID: "ef480d1a-af1a-41f5-b9a9-e128e3fd401f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:23.437982 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.437965    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef480d1a-af1a-41f5-b9a9-e128e3fd401f" (UID: "ef480d1a-af1a-41f5-b9a9-e128e3fd401f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:23.438027 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.438003    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-kube-api-access-sbwc8" (OuterVolumeSpecName: "kube-api-access-sbwc8") pod "ef480d1a-af1a-41f5-b9a9-e128e3fd401f" (UID: "ef480d1a-af1a-41f5-b9a9-e128e3fd401f"). InnerVolumeSpecName "kube-api-access-sbwc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:23.536903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.536858    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:23.536903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.536897    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:23.536903 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.536908    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwc8\" (UniqueName: \"kubernetes.io/projected/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-kube-api-access-sbwc8\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:23.537154 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.536916    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef480d1a-af1a-41f5-b9a9-e128e3fd401f-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:23.854376 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.854342    2556 generic.go:356] "Generic (PLEG): container finished" podID="ef480d1a-af1a-41f5-b9a9-e128e3fd401f" containerID="af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a" exitCode=0
Sep 16 02:46:23.854583 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.854392    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"
Sep 16 02:46:23.854583 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.854404    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs" event={"ID":"ef480d1a-af1a-41f5-b9a9-e128e3fd401f","Type":"ContainerDied","Data":"af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a"}
Sep 16 02:46:23.854583 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.854462    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs" event={"ID":"ef480d1a-af1a-41f5-b9a9-e128e3fd401f","Type":"ContainerDied","Data":"c7ad2dc58b9f009f905507c59e662edc0c81690bd1e7d78bd42fe5bcb2e40f63"}
Sep 16 02:46:23.854583 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.854483    2556 scope.go:117] "RemoveContainer" containerID="af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a"
Sep 16 02:46:23.864517 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.864498    2556 scope.go:117] "RemoveContainer" containerID="af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a"
Sep 16 02:46:23.864793 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:46:23.864774    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a\": container with ID starting with af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a not found: ID does not exist" containerID="af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a"
Sep 16 02:46:23.864845 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.864803    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a"} err="failed to get container status \"af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a\": rpc error: code = NotFound desc = could not find container \"af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a\": container with ID starting with af363321c930d3fc32fa279921eb289b1e35820898e12f47c29d20946142d20a not found: ID does not exist"
Sep 16 02:46:23.877131 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.877108    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"]
Sep 16 02:46:23.880289 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:23.880267    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a50f4-674fd9876d-d2mqs"]
Sep 16 02:46:25.113663 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:25.113634    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef480d1a-af1a-41f5-b9a9-e128e3fd401f" path="/var/lib/kubelet/pods/ef480d1a-af1a-41f5-b9a9-e128e3fd401f/volumes"
Sep 16 02:46:41.236080 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.236045    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"]
Sep 16 02:46:41.236579 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.236291    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs" podUID="e4c86d1b-f647-491c-bb73-0a75687b5641" containerName="maas-api" containerID="cri-o://9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305" gracePeriod=30
Sep 16 02:46:41.493327 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.493273    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:46:41.599442 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.599390    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7xpw\" (UniqueName: \"kubernetes.io/projected/e4c86d1b-f647-491c-bb73-0a75687b5641-kube-api-access-r7xpw\") pod \"e4c86d1b-f647-491c-bb73-0a75687b5641\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") "
Sep 16 02:46:41.599642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.599468    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c86d1b-f647-491c-bb73-0a75687b5641-openshift-service-ca\") pod \"e4c86d1b-f647-491c-bb73-0a75687b5641\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") "
Sep 16 02:46:41.599642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.599552    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-metrics-certs\") pod \"e4c86d1b-f647-491c-bb73-0a75687b5641\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") "
Sep 16 02:46:41.599642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.599590    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-maas-api-tls\") pod \"e4c86d1b-f647-491c-bb73-0a75687b5641\" (UID: \"e4c86d1b-f647-491c-bb73-0a75687b5641\") "
Sep 16 02:46:41.599949 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.599895    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c86d1b-f647-491c-bb73-0a75687b5641-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4c86d1b-f647-491c-bb73-0a75687b5641" (UID: "e4c86d1b-f647-491c-bb73-0a75687b5641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:41.602282 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.602244    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4c86d1b-f647-491c-bb73-0a75687b5641" (UID: "e4c86d1b-f647-491c-bb73-0a75687b5641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:41.602402 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.602380    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4c86d1b-f647-491c-bb73-0a75687b5641" (UID: "e4c86d1b-f647-491c-bb73-0a75687b5641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:41.602496 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.602385    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c86d1b-f647-491c-bb73-0a75687b5641-kube-api-access-r7xpw" (OuterVolumeSpecName: "kube-api-access-r7xpw") pod "e4c86d1b-f647-491c-bb73-0a75687b5641" (UID: "e4c86d1b-f647-491c-bb73-0a75687b5641"). InnerVolumeSpecName "kube-api-access-r7xpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:41.700670 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.700623    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:41.700670 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.700667    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7xpw\" (UniqueName: \"kubernetes.io/projected/e4c86d1b-f647-491c-bb73-0a75687b5641-kube-api-access-r7xpw\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:41.700877 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.700682    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c86d1b-f647-491c-bb73-0a75687b5641-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:41.700877 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.700696    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c86d1b-f647-491c-bb73-0a75687b5641-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:41.953593 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.953559    2556 generic.go:356] "Generic (PLEG): container finished" podID="e4c86d1b-f647-491c-bb73-0a75687b5641" containerID="9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305" exitCode=0
Sep 16 02:46:41.953760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.953611    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"
Sep 16 02:46:41.953760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.953642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs" event={"ID":"e4c86d1b-f647-491c-bb73-0a75687b5641","Type":"ContainerDied","Data":"9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305"}
Sep 16 02:46:41.953760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.953703    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs" event={"ID":"e4c86d1b-f647-491c-bb73-0a75687b5641","Type":"ContainerDied","Data":"15a6f507628742800dcb65182de410fac1f1682eb0aef433ae780557173272ce"}
Sep 16 02:46:41.953760 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.953728    2556 scope.go:117] "RemoveContainer" containerID="9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305"
Sep 16 02:46:41.963331 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.963314    2556 scope.go:117] "RemoveContainer" containerID="9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305"
Sep 16 02:46:41.963640 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:46:41.963622    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305\": container with ID starting with 9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305 not found: ID does not exist" containerID="9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305"
Sep 16 02:46:41.963707 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.963648    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305"} err="failed to get container status \"9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305\": rpc error: code = NotFound desc = could not find container \"9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305\": container with ID starting with 9aae674e638782f1303efa1e20bf4f3448163d72bbded59cc1da6d68a5a42305 not found: ID does not exist"
Sep 16 02:46:41.973320 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.973300    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"]
Sep 16 02:46:41.977574 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:41.977552    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe9d17-56ccc56674-fm4zs"]
Sep 16 02:46:43.114416 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.114331    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c86d1b-f647-491c-bb73-0a75687b5641" path="/var/lib/kubelet/pods/e4c86d1b-f647-491c-bb73-0a75687b5641/volumes"
Sep 16 02:46:43.275054 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.275014    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"]
Sep 16 02:46:43.275279 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.275255    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8" podUID="b6bc698c-553e-43f4-a546-6547924a5d8b" containerName="maas-api" containerID="cri-o://b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776" gracePeriod=30
Sep 16 02:46:43.536753 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.536729    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:46:43.620042 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.620011    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-metrics-certs\") pod \"b6bc698c-553e-43f4-a546-6547924a5d8b\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") "
Sep 16 02:46:43.620244 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.620066    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9lnq\" (UniqueName: \"kubernetes.io/projected/b6bc698c-553e-43f4-a546-6547924a5d8b-kube-api-access-d9lnq\") pod \"b6bc698c-553e-43f4-a546-6547924a5d8b\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") "
Sep 16 02:46:43.620244 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.620093    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bc698c-553e-43f4-a546-6547924a5d8b-openshift-service-ca\") pod \"b6bc698c-553e-43f4-a546-6547924a5d8b\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") "
Sep 16 02:46:43.620244 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.620145    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls\") pod \"b6bc698c-553e-43f4-a546-6547924a5d8b\" (UID: \"b6bc698c-553e-43f4-a546-6547924a5d8b\") "
Sep 16 02:46:43.620572 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.620545    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bc698c-553e-43f4-a546-6547924a5d8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6bc698c-553e-43f4-a546-6547924a5d8b" (UID: "b6bc698c-553e-43f4-a546-6547924a5d8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:43.622162 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.622136    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bc698c-553e-43f4-a546-6547924a5d8b-kube-api-access-d9lnq" (OuterVolumeSpecName: "kube-api-access-d9lnq") pod "b6bc698c-553e-43f4-a546-6547924a5d8b" (UID: "b6bc698c-553e-43f4-a546-6547924a5d8b"). InnerVolumeSpecName "kube-api-access-d9lnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:43.622232 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.622183    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6bc698c-553e-43f4-a546-6547924a5d8b" (UID: "b6bc698c-553e-43f4-a546-6547924a5d8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:43.622232 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.622199    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b6bc698c-553e-43f4-a546-6547924a5d8b" (UID: "b6bc698c-553e-43f4-a546-6547924a5d8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:43.720983 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.720948    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:43.720983 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.720980    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9lnq\" (UniqueName: \"kubernetes.io/projected/b6bc698c-553e-43f4-a546-6547924a5d8b-kube-api-access-d9lnq\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:43.720983 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.720990    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bc698c-553e-43f4-a546-6547924a5d8b-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:43.721222 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.721000    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6bc698c-553e-43f4-a546-6547924a5d8b-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:43.966146 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.966107    2556 generic.go:356] "Generic (PLEG): container finished" podID="b6bc698c-553e-43f4-a546-6547924a5d8b" containerID="b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776" exitCode=0
Sep 16 02:46:43.966321 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.966151    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"
Sep 16 02:46:43.966321 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.966158    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8" event={"ID":"b6bc698c-553e-43f4-a546-6547924a5d8b","Type":"ContainerDied","Data":"b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776"}
Sep 16 02:46:43.966321 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.966208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8" event={"ID":"b6bc698c-553e-43f4-a546-6547924a5d8b","Type":"ContainerDied","Data":"6f84012ab1a389a48af5284d390cb16f1cc04b2f383fe7b95022ee2d25d55ff0"}
Sep 16 02:46:43.966321 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.966233    2556 scope.go:117] "RemoveContainer" containerID="b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776"
Sep 16 02:46:43.975676 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.975655    2556 scope.go:117] "RemoveContainer" containerID="b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776"
Sep 16 02:46:43.975953 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:46:43.975934    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776\": container with ID starting with b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776 not found: ID does not exist" containerID="b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776"
Sep 16 02:46:43.976004 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.975963    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776"} err="failed to get container status \"b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776\": rpc error: code = NotFound desc = could not find container \"b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776\": container with ID starting with b74ada22744f42b5333c3be3d43693a080c8733f0e92d4dbccc2302cbdb24776 not found: ID does not exist"
Sep 16 02:46:43.986860 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.986834    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"]
Sep 16 02:46:43.990709 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:43.990684    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-10a6b9-5bf4c7768d-f44d8"]
Sep 16 02:46:45.114814 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:45.114780    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6bc698c-553e-43f4-a546-6547924a5d8b" path="/var/lib/kubelet/pods/b6bc698c-553e-43f4-a546-6547924a5d8b/volumes"
Sep 16 02:46:48.947846 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:48.947808    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"]
Sep 16 02:46:49.107886 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:49.107849    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:46:58.464700 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.464651    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"]
Sep 16 02:46:58.465215 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.464889    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr" podUID="c13ee01d-49cb-4d16-aa5a-e7313245935a" containerName="maas-api" containerID="cri-o://e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943" gracePeriod=30
Sep 16 02:46:58.725200 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.725136    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:46:58.869091 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.869059    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-metrics-certs\") pod \"c13ee01d-49cb-4d16-aa5a-e7313245935a\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") "
Sep 16 02:46:58.869091 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.869095    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn5gv\" (UniqueName: \"kubernetes.io/projected/c13ee01d-49cb-4d16-aa5a-e7313245935a-kube-api-access-sn5gv\") pod \"c13ee01d-49cb-4d16-aa5a-e7313245935a\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") "
Sep 16 02:46:58.869309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.869125    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls\") pod \"c13ee01d-49cb-4d16-aa5a-e7313245935a\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") "
Sep 16 02:46:58.869309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.869189    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13ee01d-49cb-4d16-aa5a-e7313245935a-openshift-service-ca\") pod \"c13ee01d-49cb-4d16-aa5a-e7313245935a\" (UID: \"c13ee01d-49cb-4d16-aa5a-e7313245935a\") "
Sep 16 02:46:58.869642 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.869614    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13ee01d-49cb-4d16-aa5a-e7313245935a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c13ee01d-49cb-4d16-aa5a-e7313245935a" (UID: "c13ee01d-49cb-4d16-aa5a-e7313245935a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:46:58.871217 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.871186    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c13ee01d-49cb-4d16-aa5a-e7313245935a" (UID: "c13ee01d-49cb-4d16-aa5a-e7313245935a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:58.871327 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.871225    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13ee01d-49cb-4d16-aa5a-e7313245935a-kube-api-access-sn5gv" (OuterVolumeSpecName: "kube-api-access-sn5gv") pod "c13ee01d-49cb-4d16-aa5a-e7313245935a" (UID: "c13ee01d-49cb-4d16-aa5a-e7313245935a"). InnerVolumeSpecName "kube-api-access-sn5gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:46:58.871327 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.871246    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c13ee01d-49cb-4d16-aa5a-e7313245935a" (UID: "c13ee01d-49cb-4d16-aa5a-e7313245935a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:46:58.970291 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.970257    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13ee01d-49cb-4d16-aa5a-e7313245935a-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:58.970291 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.970287    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:58.970291 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.970296    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn5gv\" (UniqueName: \"kubernetes.io/projected/c13ee01d-49cb-4d16-aa5a-e7313245935a-kube-api-access-sn5gv\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:58.970549 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:58.970305    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13ee01d-49cb-4d16-aa5a-e7313245935a-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:46:59.046317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.046230    2556 generic.go:356] "Generic (PLEG): container finished" podID="c13ee01d-49cb-4d16-aa5a-e7313245935a" containerID="e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943" exitCode=0
Sep 16 02:46:59.046317 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.046271    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"
Sep 16 02:46:59.046562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.046318    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr" event={"ID":"c13ee01d-49cb-4d16-aa5a-e7313245935a","Type":"ContainerDied","Data":"e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943"}
Sep 16 02:46:59.046562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.046368    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr" event={"ID":"c13ee01d-49cb-4d16-aa5a-e7313245935a","Type":"ContainerDied","Data":"0dd836f053dd5310b76faf441526e99e464dcfe0fa70277633f188b70fe84127"}
Sep 16 02:46:59.046562 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.046390    2556 scope.go:117] "RemoveContainer" containerID="e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943"
Sep 16 02:46:59.055926 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.055910    2556 scope.go:117] "RemoveContainer" containerID="e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943"
Sep 16 02:46:59.056213 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:46:59.056194    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943\": container with ID starting with e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943 not found: ID does not exist" containerID="e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943"
Sep 16 02:46:59.056264 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.056222    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943"} err="failed to get container status \"e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943\": rpc error: code = NotFound desc = could not find container \"e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943\": container with ID starting with e2e001958cf963f32e51e2f5139f1f1ae003fbee793e05eba3d92ca1ff3d0943 not found: ID does not exist"
Sep 16 02:46:59.066732 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.066709    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"]
Sep 16 02:46:59.070418 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.070395    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e86d9e-66fccbf5cd-g6sdr"]
Sep 16 02:46:59.121406 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:46:59.121374    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13ee01d-49cb-4d16-aa5a-e7313245935a" path="/var/lib/kubelet/pods/c13ee01d-49cb-4d16-aa5a-e7313245935a/volumes"
Sep 16 02:47:03.967171 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:03.967128    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk" podUID="6e705720-7a42-4466-8f3a-6ff07a042f0f" containerName="main" containerID="cri-o://c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603" gracePeriod=45
Sep 16 02:47:04.212423 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.212400    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:47:04.321154 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321068    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-dshm\") pod \"6e705720-7a42-4466-8f3a-6ff07a042f0f\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") "
Sep 16 02:47:04.321154 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321120    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69kk\" (UniqueName: \"kubernetes.io/projected/6e705720-7a42-4466-8f3a-6ff07a042f0f-kube-api-access-k69kk\") pod \"6e705720-7a42-4466-8f3a-6ff07a042f0f\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") "
Sep 16 02:47:04.321356 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321192    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-tmp-dir\") pod \"6e705720-7a42-4466-8f3a-6ff07a042f0f\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") "
Sep 16 02:47:04.321415 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321389    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-model-cache\") pod \"6e705720-7a42-4466-8f3a-6ff07a042f0f\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") "
Sep 16 02:47:04.321506 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321477    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-home\") pod \"6e705720-7a42-4466-8f3a-6ff07a042f0f\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") "
Sep 16 02:47:04.321506 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321479    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e705720-7a42-4466-8f3a-6ff07a042f0f" (UID: "6e705720-7a42-4466-8f3a-6ff07a042f0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:04.321608 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321576    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e705720-7a42-4466-8f3a-6ff07a042f0f-tls-certs\") pod \"6e705720-7a42-4466-8f3a-6ff07a042f0f\" (UID: \"6e705720-7a42-4466-8f3a-6ff07a042f0f\") "
Sep 16 02:47:04.321669 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321642    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e705720-7a42-4466-8f3a-6ff07a042f0f" (UID: "6e705720-7a42-4466-8f3a-6ff07a042f0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:04.321762 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321739    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-home" (OuterVolumeSpecName: "home") pod "6e705720-7a42-4466-8f3a-6ff07a042f0f" (UID: "6e705720-7a42-4466-8f3a-6ff07a042f0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:04.321959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321935    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:04.321959 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321958    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:04.322123 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.321973    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:04.323333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.323309    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-dshm" (OuterVolumeSpecName: "dshm") pod "6e705720-7a42-4466-8f3a-6ff07a042f0f" (UID: "6e705720-7a42-4466-8f3a-6ff07a042f0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:47:04.323591 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.323572    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e705720-7a42-4466-8f3a-6ff07a042f0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e705720-7a42-4466-8f3a-6ff07a042f0f" (UID: "6e705720-7a42-4466-8f3a-6ff07a042f0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:47:04.324037 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.324016    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e705720-7a42-4466-8f3a-6ff07a042f0f-kube-api-access-k69kk" (OuterVolumeSpecName: "kube-api-access-k69kk") pod "6e705720-7a42-4466-8f3a-6ff07a042f0f" (UID: "6e705720-7a42-4466-8f3a-6ff07a042f0f"). InnerVolumeSpecName "kube-api-access-k69kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:04.422713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.422675    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e705720-7a42-4466-8f3a-6ff07a042f0f-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:04.422713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.422707    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e705720-7a42-4466-8f3a-6ff07a042f0f-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:04.422713 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:04.422717    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69kk\" (UniqueName: \"kubernetes.io/projected/6e705720-7a42-4466-8f3a-6ff07a042f0f-kube-api-access-k69kk\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:05.079968 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.079936    2556 generic.go:356] "Generic (PLEG): container finished" podID="6e705720-7a42-4466-8f3a-6ff07a042f0f" containerID="c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603" exitCode=0
Sep 16 02:47:05.080389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.079977    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"
Sep 16 02:47:05.080389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.080011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk" event={"ID":"6e705720-7a42-4466-8f3a-6ff07a042f0f","Type":"ContainerDied","Data":"c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603"}
Sep 16 02:47:05.080389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.080042    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk" event={"ID":"6e705720-7a42-4466-8f3a-6ff07a042f0f","Type":"ContainerDied","Data":"be9ea64675bd46693d44e58f5579ee166c4e716853a271ff396b2c02ce695403"}
Sep 16 02:47:05.080389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.080057    2556 scope.go:117] "RemoveContainer" containerID="c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603"
Sep 16 02:47:05.088955 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.088937    2556 scope.go:117] "RemoveContainer" containerID="c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603"
Sep 16 02:47:05.089242 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:47:05.089225    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603\": container with ID starting with c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603 not found: ID does not exist" containerID="c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603"
Sep 16 02:47:05.089285 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.089251    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603"} err="failed to get container status \"c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603\": rpc error: code = NotFound desc = could not find container \"c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603\": container with ID starting with c5b394bf7400ea5d1a49d99f57167dbae2485ba57d7e8c2f2796d0367e9a3603 not found: ID does not exist"
Sep 16 02:47:05.100033 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.100010    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"]
Sep 16 02:47:05.101955 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.101934    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-10a6b9/ipp-route-10a6b9-kserve-56c99699f9-rzmqk"]
Sep 16 02:47:05.114011 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:05.113991    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e705720-7a42-4466-8f3a-6ff07a042f0f" path="/var/lib/kubelet/pods/6e705720-7a42-4466-8f3a-6ff07a042f0f/volumes"
Sep 16 02:47:42.466838 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.466790    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"]
Sep 16 02:47:42.467354 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.467045    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" podUID="8bec56eb-142b-4d14-8b0e-02fc993d116a" containerName="maas-api" containerID="cri-o://2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f" gracePeriod=30
Sep 16 02:47:42.724389 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.724326    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:47:42.882581 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.882524    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls\") pod \"8bec56eb-142b-4d14-8b0e-02fc993d116a\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") "
Sep 16 02:47:42.882773 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.882653    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hndz\" (UniqueName: \"kubernetes.io/projected/8bec56eb-142b-4d14-8b0e-02fc993d116a-kube-api-access-8hndz\") pod \"8bec56eb-142b-4d14-8b0e-02fc993d116a\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") "
Sep 16 02:47:42.882773 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.882683    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec56eb-142b-4d14-8b0e-02fc993d116a-openshift-service-ca\") pod \"8bec56eb-142b-4d14-8b0e-02fc993d116a\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") "
Sep 16 02:47:42.882773 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.882702    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-metrics-certs\") pod \"8bec56eb-142b-4d14-8b0e-02fc993d116a\" (UID: \"8bec56eb-142b-4d14-8b0e-02fc993d116a\") "
Sep 16 02:47:42.883116 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.883079    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bec56eb-142b-4d14-8b0e-02fc993d116a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bec56eb-142b-4d14-8b0e-02fc993d116a" (UID: "8bec56eb-142b-4d14-8b0e-02fc993d116a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:47:42.884795 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.884754    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8bec56eb-142b-4d14-8b0e-02fc993d116a" (UID: "8bec56eb-142b-4d14-8b0e-02fc993d116a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:47:42.884894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.884818    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bec56eb-142b-4d14-8b0e-02fc993d116a-kube-api-access-8hndz" (OuterVolumeSpecName: "kube-api-access-8hndz") pod "8bec56eb-142b-4d14-8b0e-02fc993d116a" (UID: "8bec56eb-142b-4d14-8b0e-02fc993d116a"). InnerVolumeSpecName "kube-api-access-8hndz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:47:42.884894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.884832    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bec56eb-142b-4d14-8b0e-02fc993d116a" (UID: "8bec56eb-142b-4d14-8b0e-02fc993d116a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:47:42.983660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.983573    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:42.983660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.983605    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hndz\" (UniqueName: \"kubernetes.io/projected/8bec56eb-142b-4d14-8b0e-02fc993d116a-kube-api-access-8hndz\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:42.983660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.983616    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec56eb-142b-4d14-8b0e-02fc993d116a-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:42.983660 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:42.983625    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bec56eb-142b-4d14-8b0e-02fc993d116a-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:47:43.270256 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.270157    2556 generic.go:356] "Generic (PLEG): container finished" podID="8bec56eb-142b-4d14-8b0e-02fc993d116a" containerID="2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f" exitCode=0
Sep 16 02:47:43.270256 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.270223    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"
Sep 16 02:47:43.270256 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.270240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" event={"ID":"8bec56eb-142b-4d14-8b0e-02fc993d116a","Type":"ContainerDied","Data":"2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f"}
Sep 16 02:47:43.270551 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.270296    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4" event={"ID":"8bec56eb-142b-4d14-8b0e-02fc993d116a","Type":"ContainerDied","Data":"1a45da4c5d4357460338eba84e72ea783c9310454582fc0d51424ad6c4469fab"}
Sep 16 02:47:43.270551 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.270318    2556 scope.go:117] "RemoveContainer" containerID="2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f"
Sep 16 02:47:43.279297 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.279278    2556 scope.go:117] "RemoveContainer" containerID="2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f"
Sep 16 02:47:43.279575 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:47:43.279554    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f\": container with ID starting with 2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f not found: ID does not exist" containerID="2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f"
Sep 16 02:47:43.279632 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.279582    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f"} err="failed to get container status \"2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f\": rpc error: code = NotFound desc = could not find container \"2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f\": container with ID starting with 2141287d0efb58e547361663057028496f590e95eb078c05ad395121cfb4082f not found: ID does not exist"
Sep 16 02:47:43.286192 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.286169    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"]
Sep 16 02:47:43.289605 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:43.289584    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fe974-cb767d645-wlgp4"]
Sep 16 02:47:45.113879 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:45.113849    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bec56eb-142b-4d14-8b0e-02fc993d116a" path="/var/lib/kubelet/pods/8bec56eb-142b-4d14-8b0e-02fc993d116a/volumes"
Sep 16 02:47:59.771369 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:59.771329    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"]
Sep 16 02:47:59.771976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:47:59.771600    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2" podUID="26fd1472-b966-4d6e-8f5c-984005162c5a" containerName="maas-api" containerID="cri-o://f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f" gracePeriod=30
Sep 16 02:48:00.023336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.023262    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:48:00.138387 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.138352    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-maas-api-tls\") pod \"26fd1472-b966-4d6e-8f5c-984005162c5a\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") "
Sep 16 02:48:00.138569 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.138415    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp77x\" (UniqueName: \"kubernetes.io/projected/26fd1472-b966-4d6e-8f5c-984005162c5a-kube-api-access-kp77x\") pod \"26fd1472-b966-4d6e-8f5c-984005162c5a\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") "
Sep 16 02:48:00.138569 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.138464    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd1472-b966-4d6e-8f5c-984005162c5a-openshift-service-ca\") pod \"26fd1472-b966-4d6e-8f5c-984005162c5a\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") "
Sep 16 02:48:00.138569 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.138512    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-metrics-certs\") pod \"26fd1472-b966-4d6e-8f5c-984005162c5a\" (UID: \"26fd1472-b966-4d6e-8f5c-984005162c5a\") "
Sep 16 02:48:00.138843 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.138812    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fd1472-b966-4d6e-8f5c-984005162c5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26fd1472-b966-4d6e-8f5c-984005162c5a" (UID: "26fd1472-b966-4d6e-8f5c-984005162c5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:48:00.140600 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.140571    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26fd1472-b966-4d6e-8f5c-984005162c5a" (UID: "26fd1472-b966-4d6e-8f5c-984005162c5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:48:00.140698 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.140648    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fd1472-b966-4d6e-8f5c-984005162c5a-kube-api-access-kp77x" (OuterVolumeSpecName: "kube-api-access-kp77x") pod "26fd1472-b966-4d6e-8f5c-984005162c5a" (UID: "26fd1472-b966-4d6e-8f5c-984005162c5a"). InnerVolumeSpecName "kube-api-access-kp77x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:48:00.140794 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.140767    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26fd1472-b966-4d6e-8f5c-984005162c5a" (UID: "26fd1472-b966-4d6e-8f5c-984005162c5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:48:00.239622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.239588    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:00.239622 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.239621    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp77x\" (UniqueName: \"kubernetes.io/projected/26fd1472-b966-4d6e-8f5c-984005162c5a-kube-api-access-kp77x\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:00.239833 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.239722    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd1472-b966-4d6e-8f5c-984005162c5a-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:00.239833 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.239742    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26fd1472-b966-4d6e-8f5c-984005162c5a-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:00.358458 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.358342    2556 generic.go:356] "Generic (PLEG): container finished" podID="26fd1472-b966-4d6e-8f5c-984005162c5a" containerID="f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f" exitCode=0
Sep 16 02:48:00.358458 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.358381    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2" event={"ID":"26fd1472-b966-4d6e-8f5c-984005162c5a","Type":"ContainerDied","Data":"f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f"}
Sep 16 02:48:00.358458 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.358385    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"
Sep 16 02:48:00.358458 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.358420    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2" event={"ID":"26fd1472-b966-4d6e-8f5c-984005162c5a","Type":"ContainerDied","Data":"a0410f6686458070b8b38032e42c8d3008f39c27671e44c810290289d46e2b8b"}
Sep 16 02:48:00.358742 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.358472    2556 scope.go:117] "RemoveContainer" containerID="f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f"
Sep 16 02:48:00.367703 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.367683    2556 scope.go:117] "RemoveContainer" containerID="f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f"
Sep 16 02:48:00.367984 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:48:00.367965    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f\": container with ID starting with f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f not found: ID does not exist" containerID="f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f"
Sep 16 02:48:00.368025 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.367995    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f"} err="failed to get container status \"f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f\": rpc error: code = NotFound desc = could not find container \"f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f\": container with ID starting with f5e661f4efa6258660716b1af2434a23cdfb90bfc803638af9a9487a7b00b16f not found: ID does not exist"
Sep 16 02:48:00.378104 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.378061    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"]
Sep 16 02:48:00.380276 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:00.380251    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db1ed7-5fcc6dd668-hlps2"]
Sep 16 02:48:01.114963 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:01.114929    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fd1472-b966-4d6e-8f5c-984005162c5a" path="/var/lib/kubelet/pods/26fd1472-b966-4d6e-8f5c-984005162c5a/volumes"
Sep 16 02:48:09.107638 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:09.107597    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:48:17.207199 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.207107    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"]
Sep 16 02:48:17.207771 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.207345    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd" podUID="fe9630f5-76ac-4144-9302-3626ad47d2d4" containerName="maas-api" containerID="cri-o://3d7d83abc6435ccf6cd422cb2dd1bdde5ce2922c5ea243816251b36821173fdc" gracePeriod=30
Sep 16 02:48:17.443227 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.443193    2556 generic.go:356] "Generic (PLEG): container finished" podID="fe9630f5-76ac-4144-9302-3626ad47d2d4" containerID="3d7d83abc6435ccf6cd422cb2dd1bdde5ce2922c5ea243816251b36821173fdc" exitCode=0
Sep 16 02:48:17.443361 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.443258    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd" event={"ID":"fe9630f5-76ac-4144-9302-3626ad47d2d4","Type":"ContainerDied","Data":"3d7d83abc6435ccf6cd422cb2dd1bdde5ce2922c5ea243816251b36821173fdc"}
Sep 16 02:48:17.443361 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.443317    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd" event={"ID":"fe9630f5-76ac-4144-9302-3626ad47d2d4","Type":"ContainerDied","Data":"9685310ad17671923b15cd2d86035895dcb86c8a198e595dbaec722b303dd50d"}
Sep 16 02:48:17.443361 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.443334    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9685310ad17671923b15cd2d86035895dcb86c8a198e595dbaec722b303dd50d"
Sep 16 02:48:17.451217 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.451196    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:48:17.496208 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.496113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-metrics-certs\") pod \"fe9630f5-76ac-4144-9302-3626ad47d2d4\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") "
Sep 16 02:48:17.496208 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.496188    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbhc5\" (UniqueName: \"kubernetes.io/projected/fe9630f5-76ac-4144-9302-3626ad47d2d4-kube-api-access-fbhc5\") pod \"fe9630f5-76ac-4144-9302-3626ad47d2d4\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") "
Sep 16 02:48:17.496469 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.496232    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-maas-api-tls\") pod \"fe9630f5-76ac-4144-9302-3626ad47d2d4\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") "
Sep 16 02:48:17.496469 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.496287    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe9630f5-76ac-4144-9302-3626ad47d2d4-openshift-service-ca\") pod \"fe9630f5-76ac-4144-9302-3626ad47d2d4\" (UID: \"fe9630f5-76ac-4144-9302-3626ad47d2d4\") "
Sep 16 02:48:17.496981 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.496947    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9630f5-76ac-4144-9302-3626ad47d2d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe9630f5-76ac-4144-9302-3626ad47d2d4" (UID: "fe9630f5-76ac-4144-9302-3626ad47d2d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 02:48:17.498727 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.498697    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe9630f5-76ac-4144-9302-3626ad47d2d4" (UID: "fe9630f5-76ac-4144-9302-3626ad47d2d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:48:17.499396 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.499375    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe9630f5-76ac-4144-9302-3626ad47d2d4" (UID: "fe9630f5-76ac-4144-9302-3626ad47d2d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:48:17.499666 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.499636    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9630f5-76ac-4144-9302-3626ad47d2d4-kube-api-access-fbhc5" (OuterVolumeSpecName: "kube-api-access-fbhc5") pod "fe9630f5-76ac-4144-9302-3626ad47d2d4" (UID: "fe9630f5-76ac-4144-9302-3626ad47d2d4"). InnerVolumeSpecName "kube-api-access-fbhc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:48:17.597266 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.597223    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-metrics-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:17.597266 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.597261    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhc5\" (UniqueName: \"kubernetes.io/projected/fe9630f5-76ac-4144-9302-3626ad47d2d4-kube-api-access-fbhc5\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:17.597266 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.597271    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe9630f5-76ac-4144-9302-3626ad47d2d4-maas-api-tls\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:17.597266 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:17.597279    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe9630f5-76ac-4144-9302-3626ad47d2d4-openshift-service-ca\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:48:18.448377 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:18.448344    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"
Sep 16 02:48:18.468775 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:18.468747    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"]
Sep 16 02:48:18.472176 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:18.472152    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-eee708-7cff54cdf4-vwtdd"]
Sep 16 02:48:19.114047 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:19.114013    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe9630f5-76ac-4144-9302-3626ad47d2d4" path="/var/lib/kubelet/pods/fe9630f5-76ac-4144-9302-3626ad47d2d4/volumes"
Sep 16 02:48:21.095169 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:21.095138    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:48:21.100368 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:21.100346    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:48:21.312807 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:48:21.312769    2556 scope.go:117] "RemoveContainer" containerID="3d7d83abc6435ccf6cd422cb2dd1bdde5ce2922c5ea243816251b36821173fdc"
Sep 16 02:49:23.108336 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:49:23.108300    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:50:45.108295 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:50:45.108247    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:51:01.764922 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:01.764888    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"]
Sep 16 02:51:01.765377 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:01.765095    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j" podUID="0fabda73-1c04-4032-8a25-2ab7351f572e" containerName="manager" containerID="cri-o://86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c" gracePeriod=10
Sep 16 02:51:02.215128 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.215099    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:51:02.255732 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.255694    2556 generic.go:356] "Generic (PLEG): container finished" podID="0fabda73-1c04-4032-8a25-2ab7351f572e" containerID="86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c" exitCode=0
Sep 16 02:51:02.255880 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.255731    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j" event={"ID":"0fabda73-1c04-4032-8a25-2ab7351f572e","Type":"ContainerDied","Data":"86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c"}
Sep 16 02:51:02.255880 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.255793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j" event={"ID":"0fabda73-1c04-4032-8a25-2ab7351f572e","Type":"ContainerDied","Data":"e4091772abdb5b5db12599641f1545a89d809373bf55229cdb15be8ef500ab07"}
Sep 16 02:51:02.255880 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.255738    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"
Sep 16 02:51:02.255880 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.255824    2556 scope.go:117] "RemoveContainer" containerID="86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c"
Sep 16 02:51:02.265521 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.265503    2556 scope.go:117] "RemoveContainer" containerID="86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c"
Sep 16 02:51:02.265792 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:51:02.265765    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c\": container with ID starting with 86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c not found: ID does not exist" containerID="86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c"
Sep 16 02:51:02.265888 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.265798    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c"} err="failed to get container status \"86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c\": rpc error: code = NotFound desc = could not find container \"86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c\": container with ID starting with 86bb676019e979c3039c5675af2b1a776b5080ccad27d26fd9bd62f6cc6ca52c not found: ID does not exist"
Sep 16 02:51:02.338309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.338234    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fabda73-1c04-4032-8a25-2ab7351f572e-extensions-socket-volume\") pod \"0fabda73-1c04-4032-8a25-2ab7351f572e\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") "
Sep 16 02:51:02.338309 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.338273    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7jks\" (UniqueName: \"kubernetes.io/projected/0fabda73-1c04-4032-8a25-2ab7351f572e-kube-api-access-s7jks\") pod \"0fabda73-1c04-4032-8a25-2ab7351f572e\" (UID: \"0fabda73-1c04-4032-8a25-2ab7351f572e\") "
Sep 16 02:51:02.338710 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.338671    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fabda73-1c04-4032-8a25-2ab7351f572e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0fabda73-1c04-4032-8a25-2ab7351f572e" (UID: "0fabda73-1c04-4032-8a25-2ab7351f572e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:51:02.340258 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.340236    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fabda73-1c04-4032-8a25-2ab7351f572e-kube-api-access-s7jks" (OuterVolumeSpecName: "kube-api-access-s7jks") pod "0fabda73-1c04-4032-8a25-2ab7351f572e" (UID: "0fabda73-1c04-4032-8a25-2ab7351f572e"). InnerVolumeSpecName "kube-api-access-s7jks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:51:02.439061 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.439036    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0fabda73-1c04-4032-8a25-2ab7351f572e-extensions-socket-volume\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:51:02.439061 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.439060    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jks\" (UniqueName: \"kubernetes.io/projected/0fabda73-1c04-4032-8a25-2ab7351f572e-kube-api-access-s7jks\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:51:02.576186 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.576155    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"]
Sep 16 02:51:02.579976 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:02.579951    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xhn4j"]
Sep 16 02:51:03.113153 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:51:03.113117    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fabda73-1c04-4032-8a25-2ab7351f572e" path="/var/lib/kubelet/pods/0fabda73-1c04-4032-8a25-2ab7351f572e/volumes"
Sep 16 02:52:02.557352 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557322    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"]
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557718    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fabda73-1c04-4032-8a25-2ab7351f572e" containerName="manager"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557731    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fabda73-1c04-4032-8a25-2ab7351f572e" containerName="manager"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557741    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e705720-7a42-4466-8f3a-6ff07a042f0f" containerName="main"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557748    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e705720-7a42-4466-8f3a-6ff07a042f0f" containerName="main"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557766    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9630f5-76ac-4144-9302-3626ad47d2d4" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557774    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9630f5-76ac-4144-9302-3626ad47d2d4" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557786    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c86d1b-f647-491c-bb73-0a75687b5641" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557794    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c86d1b-f647-491c-bb73-0a75687b5641" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557807    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13ee01d-49cb-4d16-aa5a-e7313245935a" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557815    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13ee01d-49cb-4d16-aa5a-e7313245935a" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557824    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd93203-0537-4f70-aa67-b972547f824d" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557832    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd93203-0537-4f70-aa67-b972547f824d" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557847    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8fac89a-bb94-4cc6-91df-8d22b5513f3d" containerName="maas-api"
Sep 16 02:52:02.557858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557856    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8fac89a-bb94-4cc6-91df-8d22b5513f3d" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557870    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef480d1a-af1a-41f5-b9a9-e128e3fd401f" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557880    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef480d1a-af1a-41f5-b9a9-e128e3fd401f" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557894    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bec56eb-142b-4d14-8b0e-02fc993d116a" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557902    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bec56eb-142b-4d14-8b0e-02fc993d116a" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557921    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fd1472-b966-4d6e-8f5c-984005162c5a" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557929    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fd1472-b966-4d6e-8f5c-984005162c5a" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557937    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bc698c-553e-43f4-a546-6547924a5d8b" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.557945    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bc698c-553e-43f4-a546-6547924a5d8b" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558045    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bc698c-553e-43f4-a546-6547924a5d8b" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558056    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bec56eb-142b-4d14-8b0e-02fc993d116a" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558068    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c86d1b-f647-491c-bb73-0a75687b5641" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558078    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fd1472-b966-4d6e-8f5c-984005162c5a" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558087    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fabda73-1c04-4032-8a25-2ab7351f572e" containerName="manager"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558095    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9630f5-76ac-4144-9302-3626ad47d2d4" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558103    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd93203-0537-4f70-aa67-b972547f824d" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558110    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e705720-7a42-4466-8f3a-6ff07a042f0f" containerName="main"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558118    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13ee01d-49cb-4d16-aa5a-e7313245935a" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558126    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8fac89a-bb94-4cc6-91df-8d22b5513f3d" containerName="maas-api"
Sep 16 02:52:02.558333 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.558133    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef480d1a-af1a-41f5-b9a9-e128e3fd401f" containerName="maas-api"
Sep 16 02:52:02.561717 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.561699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:02.564206 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.564184    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vbcj6\""
Sep 16 02:52:02.574173 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.574147    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"]
Sep 16 02:52:02.664183 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.664149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5fda9f9-f65a-42e0-8c45-021658161064-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4q68p\" (UID: \"b5fda9f9-f65a-42e0-8c45-021658161064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:02.664351 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.664235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wv5\" (UniqueName: \"kubernetes.io/projected/b5fda9f9-f65a-42e0-8c45-021658161064-kube-api-access-m8wv5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4q68p\" (UID: \"b5fda9f9-f65a-42e0-8c45-021658161064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:02.765248 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.765212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wv5\" (UniqueName: \"kubernetes.io/projected/b5fda9f9-f65a-42e0-8c45-021658161064-kube-api-access-m8wv5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4q68p\" (UID: \"b5fda9f9-f65a-42e0-8c45-021658161064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:02.765455 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.765271    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5fda9f9-f65a-42e0-8c45-021658161064-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4q68p\" (UID: \"b5fda9f9-f65a-42e0-8c45-021658161064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:02.765625 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.765603    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5fda9f9-f65a-42e0-8c45-021658161064-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4q68p\" (UID: \"b5fda9f9-f65a-42e0-8c45-021658161064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:02.775979 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.775953    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wv5\" (UniqueName: \"kubernetes.io/projected/b5fda9f9-f65a-42e0-8c45-021658161064-kube-api-access-m8wv5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4q68p\" (UID: \"b5fda9f9-f65a-42e0-8c45-021658161064\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:02.872343 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:02.872247    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:03.155008 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:03.154985    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"]
Sep 16 02:52:03.156985 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:52:03.156959    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fda9f9_f65a_42e0_8c45_021658161064.slice/crio-4514c6e905c5d81cc853ca6dddde8559cf868664e6b92fd9ddcc803bcd5321a3 WatchSource:0}: Error finding container 4514c6e905c5d81cc853ca6dddde8559cf868664e6b92fd9ddcc803bcd5321a3: Status 404 returned error can't find the container with id 4514c6e905c5d81cc853ca6dddde8559cf868664e6b92fd9ddcc803bcd5321a3
Sep 16 02:52:03.564057 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:03.564018    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p" event={"ID":"b5fda9f9-f65a-42e0-8c45-021658161064","Type":"ContainerStarted","Data":"b6e011b793e7b509d29c123a4d70479397a0c624500c763f44f35995c8a561df"}
Sep 16 02:52:03.564478 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:03.564080    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p" event={"ID":"b5fda9f9-f65a-42e0-8c45-021658161064","Type":"ContainerStarted","Data":"4514c6e905c5d81cc853ca6dddde8559cf868664e6b92fd9ddcc803bcd5321a3"}
Sep 16 02:52:03.564478 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:03.564155    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:03.582247 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:03.582190    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p" podStartSLOduration=1.582177701 podStartE2EDuration="1.582177701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:52:03.582083859 +0000 UTC m=+2023.012145246" watchObservedRunningTime="2026-09-16 02:52:03.582177701 +0000 UTC m=+2023.012239086"
Sep 16 02:52:04.108200 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:04.108169    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:52:14.571543 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:14.571505    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4q68p"
Sep 16 02:52:59.381459 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.381407    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"]
Sep 16 02:52:59.384662 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.384642    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.387722 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.387699    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c0541850-kserve-self-signed-certs\""
Sep 16 02:52:59.406243 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.406213    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"]
Sep 16 02:52:59.484854 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.484819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-model-cache\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.485037 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.484861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tls-certs\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.485037 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.484885    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tmp-dir\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.485037 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.484933    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-dshm\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.485143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.485041    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6dg\" (UniqueName: \"kubernetes.io/projected/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-kube-api-access-dq6dg\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.485143 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.485072    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-home\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586276 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586241    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-model-cache\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586439 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586284    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tls-certs\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586439 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586324    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tmp-dir\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586439 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586355    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-dshm\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586439 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6dg\" (UniqueName: \"kubernetes.io/projected/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-kube-api-access-dq6dg\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586675 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586479    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-home\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586778 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586752    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tmp-dir\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586832 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586792    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-model-cache\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.586888 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.586871    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-home\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.588669 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.588650    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-dshm\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.588966 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.588946    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tls-certs\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.593423 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.593400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6dg\" (UniqueName: \"kubernetes.io/projected/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-kube-api-access-dq6dg\") pod \"e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") " pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.693657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.693622    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:52:59.966197 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:52:59.966169    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"]
Sep 16 02:52:59.968846 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:52:59.968817    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcb7ca01_b102_4a8e_9d18_5da6a57b8f4b.slice/crio-40c27015b9ebd38b1cd41191b43ac118e83eb32166040400039efc9822247132 WatchSource:0}: Error finding container 40c27015b9ebd38b1cd41191b43ac118e83eb32166040400039efc9822247132: Status 404 returned error can't find the container with id 40c27015b9ebd38b1cd41191b43ac118e83eb32166040400039efc9822247132
Sep 16 02:53:00.846894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:00.846855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn" event={"ID":"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b","Type":"ContainerStarted","Data":"ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278"}
Sep 16 02:53:00.846894 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:00.846901    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn" event={"ID":"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b","Type":"ContainerStarted","Data":"40c27015b9ebd38b1cd41191b43ac118e83eb32166040400039efc9822247132"}
Sep 16 02:53:00.863048 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:00.862997    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn" podStartSLOduration=1.8629812399999999 podStartE2EDuration="1.86298124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 02:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:53:00.861025846 +0000 UTC m=+2080.291087259" watchObservedRunningTime="2026-09-16 02:53:00.86298124 +0000 UTC m=+2080.293042626"
Sep 16 02:53:09.694463 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:09.694409    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:53:09.694935 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:09.694488    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:53:09.707466 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:09.707422    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:53:09.904463 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:09.904416    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:53:13.108339 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:13.108292    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:53:21.126941 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:21.126912    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:53:21.133954 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:21.133926    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:53:41.758191 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:41.758154    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"]
Sep 16 02:53:56.777440 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:56.777388    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn" podUID="bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" containerName="main" containerID="cri-o://ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278" gracePeriod=45
Sep 16 02:53:57.032733 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.032673    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:53:57.096198 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096160    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-model-cache\") pod \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") "
Sep 16 02:53:57.096381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096205    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-dshm\") pod \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") "
Sep 16 02:53:57.096381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096262    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq6dg\" (UniqueName: \"kubernetes.io/projected/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-kube-api-access-dq6dg\") pod \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") "
Sep 16 02:53:57.096381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096295    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-home\") pod \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") "
Sep 16 02:53:57.096381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096342    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tls-certs\") pod \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") "
Sep 16 02:53:57.096381 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096377    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tmp-dir\") pod \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\" (UID: \"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b\") "
Sep 16 02:53:57.096712 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096523    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" (UID: "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:57.096774 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096758    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-model-cache\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:53:57.096829 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096766    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" (UID: "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:57.096965 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.096941    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-home" (OuterVolumeSpecName: "home") pod "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" (UID: "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:57.098650 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.098627    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-dshm" (OuterVolumeSpecName: "dshm") pod "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" (UID: "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 02:53:57.098735 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.098627    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" (UID: "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 02:53:57.098853 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.098830    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-kube-api-access-dq6dg" (OuterVolumeSpecName: "kube-api-access-dq6dg") pod "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" (UID: "bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b"). InnerVolumeSpecName "kube-api-access-dq6dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 02:53:57.134485 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.134453    2556 generic.go:356] "Generic (PLEG): container finished" podID="bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" containerID="ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278" exitCode=0
Sep 16 02:53:57.134643 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.134496    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"
Sep 16 02:53:57.134643 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.134529    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn" event={"ID":"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b","Type":"ContainerDied","Data":"ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278"}
Sep 16 02:53:57.134643 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.134569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn" event={"ID":"bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b","Type":"ContainerDied","Data":"40c27015b9ebd38b1cd41191b43ac118e83eb32166040400039efc9822247132"}
Sep 16 02:53:57.134643 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.134586    2556 scope.go:117] "RemoveContainer" containerID="ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278"
Sep 16 02:53:57.144694 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.144676    2556 scope.go:117] "RemoveContainer" containerID="ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278"
Sep 16 02:53:57.144939 ip-10-0-143-48 kubenswrapper[2556]: E0916 02:53:57.144923    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278\": container with ID starting with ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278 not found: ID does not exist" containerID="ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278"
Sep 16 02:53:57.145016 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.144945    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278"} err="failed to get container status \"ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278\": rpc error: code = NotFound desc = could not find container \"ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278\": container with ID starting with ae96296d34248ae0654eac70e99c70e1459ab466d82483a8d18b906f21bf4278 not found: ID does not exist"
Sep 16 02:53:57.150805 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.150782    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"]
Sep 16 02:53:57.153692 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.153673    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c0541850-kserve-5c969b6447-q6jfn"]
Sep 16 02:53:57.197315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.197281    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6dg\" (UniqueName: \"kubernetes.io/projected/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-kube-api-access-dq6dg\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:53:57.197315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.197306    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-home\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:53:57.197315 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.197317    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tls-certs\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:53:57.197540 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.197342    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-tmp-dir\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:53:57.197540 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:57.197350    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b-dshm\") on node \"ip-10-0-143-48.ec2.internal\" DevicePath \"\""
Sep 16 02:53:59.114066 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:53:59.114035    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" path="/var/lib/kubelet/pods/bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b/volumes"
Sep 16 02:54:32.107727 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:54:32.107691    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:55:53.108021 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:55:53.107937    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:57:09.108294 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:09.108251    2556 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-5d7dbbd56b-p6mx9" secret="" err="secret \"payload-processing-dockercfg-7v8jt\" not found"
Sep 16 02:57:34.388455 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:34.388363    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-plfnq_9b83e697-837d-4c0e-b454-767d6743f7ab/global-pull-secret-syncer/0.log"
Sep 16 02:57:34.509875 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:34.509836    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tpdvn_8ea3ca72-53a1-456c-85f1-8750d3ac07b1/konnectivity-agent/0.log"
Sep 16 02:57:34.578241 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:34.578205    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-48.ec2.internal_c5e07eb16b748b05b653e47f2fc353a6/haproxy/0.log"
Sep 16 02:57:39.608765 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:39.608723    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p4x6c_ea5dc572-039c-4628-9d52-442525df6346/manager/0.log"
Sep 16 02:57:39.631095 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:39.631068    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-gxhc5_3bad8a8b-9834-47c6-ac7c-6c5479a9e56d/manager/0.log"
Sep 16 02:57:39.668446 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:39.668412    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-797p4_d7430254-3fe5-44c1-b21f-67a26c46bff8/kuadrant-console-plugin/0.log"
Sep 16 02:57:39.711798 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:39.711768    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-4q68p_b5fda9f9-f65a-42e0-8c45-021658161064/manager/0.log"
Sep 16 02:57:39.820443 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:39.820387    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-5r7qc_30aaa2da-a0fe-45dd-99d0-f47c36b89760/manager/0.log"
Sep 16 02:57:41.338275 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:41.338232    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-4skl4_8fb91d7c-c3a9-40f3-89fa-8aac191a07ef/cluster-monitoring-operator/0.log"
Sep 16 02:57:41.689301 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:41.689264    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5q6s_ef70daa3-7579-406f-ab26-ed24cb298c95/node-exporter/0.log"
Sep 16 02:57:41.713557 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:41.713528    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5q6s_ef70daa3-7579-406f-ab26-ed24cb298c95/kube-rbac-proxy/0.log"
Sep 16 02:57:41.733344 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:41.733317    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5q6s_ef70daa3-7579-406f-ab26-ed24cb298c95/init-textfile/0.log"
Sep 16 02:57:43.717736 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:43.717704    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-rnf47_512c9eff-f1e7-4507-9d75-fcaf212112e4/networking-console-plugin/0.log"
Sep 16 02:57:44.333051 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:44.333021    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/1.log"
Sep 16 02:57:44.340529 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:44.340500    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mgds9_97829f87-c6a8-4a47-a358-df8b7eb261dd/console-operator/2.log"
Sep 16 02:57:45.065382 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.065347    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"]
Sep 16 02:57:45.065975 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.065952    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" containerName="main"
Sep 16 02:57:45.066065 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.065977    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" containerName="main"
Sep 16 02:57:45.066123 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.066083    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb7ca01-b102-4a8e-9d18-5da6a57b8f4b" containerName="main"
Sep 16 02:57:45.069288 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.069266    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.071834 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.071813    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fv4vf\"/\"default-dockercfg-nxld2\""
Sep 16 02:57:45.071937 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.071831    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fv4vf\"/\"openshift-service-ca.crt\""
Sep 16 02:57:45.072706 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.072687    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fv4vf\"/\"kube-root-ca.crt\""
Sep 16 02:57:45.077936 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.077913    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"]
Sep 16 02:57:45.163443 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.163403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzcc\" (UniqueName: \"kubernetes.io/projected/81ca632d-a175-4fb4-b8be-6bda11f3b930-kube-api-access-nhzcc\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.163626 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.163503    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-sys\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.163626 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.163523    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-lib-modules\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.163626 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.163588    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-podres\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.163777 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.163720    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-proc\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264242 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-podres\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264407 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264253    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-proc\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264407 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264297    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzcc\" (UniqueName: \"kubernetes.io/projected/81ca632d-a175-4fb4-b8be-6bda11f3b930-kube-api-access-nhzcc\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264407 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264330    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-sys\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264407 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-lib-modules\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264407 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264383    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-podres\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-proc\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264452    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-sys\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.264657 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.264530    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81ca632d-a175-4fb4-b8be-6bda11f3b930-lib-modules\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.271261 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.271243    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzcc\" (UniqueName: \"kubernetes.io/projected/81ca632d-a175-4fb4-b8be-6bda11f3b930-kube-api-access-nhzcc\") pod \"perf-node-gather-daemonset-6wnld\" (UID: \"81ca632d-a175-4fb4-b8be-6bda11f3b930\") " pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.380527 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.380418    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:45.603037 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.603009    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-8qh6p_d29dc62f-9e58-48a5-b87f-074fe8f27b44/volume-data-source-validator/0.log"
Sep 16 02:57:45.648870 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:45.648846    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"]
Sep 16 02:57:45.651650 ip-10-0-143-48 kubenswrapper[2556]: W0916 02:57:45.651620    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod81ca632d_a175_4fb4_b8be_6bda11f3b930.slice/crio-fb19001049ba4521305e6147e2e5eb4866e3c853b421f7f9e6e63987fbb11892 WatchSource:0}: Error finding container fb19001049ba4521305e6147e2e5eb4866e3c853b421f7f9e6e63987fbb11892: Status 404 returned error can't find the container with id fb19001049ba4521305e6147e2e5eb4866e3c853b421f7f9e6e63987fbb11892
Sep 16 02:57:46.257580 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:46.257545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld" event={"ID":"81ca632d-a175-4fb4-b8be-6bda11f3b930","Type":"ContainerStarted","Data":"a149c9e1ddb0ccdc5bff49b4bc609dabaf0c11e1f5ffbb59ecbf190ddd7fe1c3"}
Sep 16 02:57:46.258011 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:46.257599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld" event={"ID":"81ca632d-a175-4fb4-b8be-6bda11f3b930","Type":"ContainerStarted","Data":"fb19001049ba4521305e6147e2e5eb4866e3c853b421f7f9e6e63987fbb11892"}
Sep 16 02:57:46.258011 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:46.257627    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:46.272484 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:46.272412    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld" podStartSLOduration=1.272393904 podStartE2EDuration="1.272393904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 02:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 02:57:46.270111091 +0000 UTC m=+2365.700172476" watchObservedRunningTime="2026-09-16 02:57:46.272393904 +0000 UTC m=+2365.702455289"
Sep 16 02:57:46.766202 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:46.766164    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcckb_7fe7d06d-b82f-407c-9b61-73189c034507/dns/0.log"
Sep 16 02:57:46.786255 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:46.786224    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcckb_7fe7d06d-b82f-407c-9b61-73189c034507/kube-rbac-proxy/0.log"
Sep 16 02:57:46.893874 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:46.893846    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jn7jj_5b5ea2d3-a6ae-4089-8204-d7422983fe38/dns-node-resolver/0.log"
Sep 16 02:57:47.523947 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:47.523921    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9w9ds_6e60ad12-0057-402f-b9fb-56ffa04e13a0/node-ca/0.log"
Sep 16 02:57:48.796974 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:48.796946    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f86d6bc5-9c7x6_046a8542-6312-4e89-b4be-350a31a37f68/kube-auth-proxy/0.log"
Sep 16 02:57:48.896943 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:48.896901    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-p6mx9_c9582f82-fc4a-448c-99e4-4fc2fc2d118d/payload-processing/0.log"
Sep 16 02:57:48.923178 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:48.923154    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9bc99d698-k2qmx_d501b79f-f7db-41af-93d6-ae81d067cb5d/router/0.log"
Sep 16 02:57:49.964027 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:49.963998    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gpkxq_7815c4f9-893e-44ba-a7dd-3fcb75aa0ab6/serve-healthcheck-canary/0.log"
Sep 16 02:57:50.456645 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:50.456604    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hs4wt_177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42/insights-operator/0.log"
Sep 16 02:57:50.457074 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:50.457037    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hs4wt_177bd2fb-c14b-4d9e-aae0-a92b2dfe9f42/insights-operator/1.log"
Sep 16 02:57:50.633493 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:50.633459    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrb6d_2a5cf79b-75fe-4673-8c10-7ef8c1499fbe/kube-rbac-proxy/0.log"
Sep 16 02:57:50.653508 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:50.653401    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrb6d_2a5cf79b-75fe-4673-8c10-7ef8c1499fbe/exporter/0.log"
Sep 16 02:57:50.677181 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:50.677148    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrb6d_2a5cf79b-75fe-4673-8c10-7ef8c1499fbe/extractor/0.log"
Sep 16 02:57:52.273054 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:52.273026    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fv4vf/perf-node-gather-daemonset-6wnld"
Sep 16 02:57:52.728142 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:52.728110    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5dbb7f5f95-zwblq_c6f201a9-82a7-4833-bf0b-1e704d0ae94e/manager/0.log"
Sep 16 02:57:53.132612 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:53.132533    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mbp22_a513ea27-fef0-43f9-96c8-8ca746e29a86/manager/0.log"
Sep 16 02:57:53.474648 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:53.474614    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599b75f66d-j2kgm_64e8af4d-013a-411d-a3e7-f6d501b6d545/manager/0.log"
Sep 16 02:57:53.491645 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:53.491624    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599b75f66d-j2kgm_64e8af4d-013a-411d-a3e7-f6d501b6d545/copy-manifests/0.log"
Sep 16 02:57:55.645188 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:55.645158    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f86d6bc5-9c7x6_046a8542-6312-4e89-b4be-350a31a37f68/kube-auth-proxy/0.log"
Sep 16 02:57:55.704783 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:55.704752    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-p6mx9_c9582f82-fc4a-448c-99e4-4fc2fc2d118d/payload-processing/0.log"
Sep 16 02:57:55.722010 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:55.721980    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9bc99d698-k2qmx_d501b79f-f7db-41af-93d6-ae81d067cb5d/router/0.log"
Sep 16 02:57:57.592492 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:57.592461    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f86d6bc5-9c7x6_046a8542-6312-4e89-b4be-350a31a37f68/kube-auth-proxy/0.log"
Sep 16 02:57:57.657614 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:57.657582    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5d7dbbd56b-p6mx9_c9582f82-fc4a-448c-99e4-4fc2fc2d118d/payload-processing/0.log"
Sep 16 02:57:57.676063 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:57:57.676036    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9bc99d698-k2qmx_d501b79f-f7db-41af-93d6-ae81d067cb5d/router/0.log"
Sep 16 02:58:00.256834 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:00.256787    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vhp4p_52249ffa-3ecf-443a-a076-1f1cdbab07cc/migrator/0.log"
Sep 16 02:58:00.276943 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:00.276913    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vhp4p_52249ffa-3ecf-443a-a076-1f1cdbab07cc/graceful-termination/0.log"
Sep 16 02:58:00.632994 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:00.632894    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5gnj8_dd5bdaad-9ef0-499f-a136-fefde5f6f7d2/kube-storage-version-migrator-operator/1.log"
Sep 16 02:58:00.634624 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:00.634596    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-5gnj8_dd5bdaad-9ef0-499f-a136-fefde5f6f7d2/kube-storage-version-migrator-operator/0.log"
Sep 16 02:58:01.923322 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:01.923290    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gdff9_2c014add-556b-46aa-a326-69b9bad2ed52/kube-multus-additional-cni-plugins/0.log"
Sep 16 02:58:01.943544 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:01.943517    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gdff9_2c014add-556b-46aa-a326-69b9bad2ed52/egress-router-binary-copy/0.log"
Sep 16 02:58:01.964676 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:01.964649    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gdff9_2c014add-556b-46aa-a326-69b9bad2ed52/cni-plugins/0.log"
Sep 16 02:58:01.985858 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:01.985839    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gdff9_2c014add-556b-46aa-a326-69b9bad2ed52/bond-cni-plugin/0.log"
Sep 16 02:58:02.005982 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:02.005959    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gdff9_2c014add-556b-46aa-a326-69b9bad2ed52/routeoverride-cni/0.log"
Sep 16 02:58:02.025758 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:02.025731    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gdff9_2c014add-556b-46aa-a326-69b9bad2ed52/whereabouts-cni-bincopy/0.log"
Sep 16 02:58:02.044811 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:02.044792    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gdff9_2c014add-556b-46aa-a326-69b9bad2ed52/whereabouts-cni/0.log"
Sep 16 02:58:02.239806 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:02.239726    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bzlfw_7ab38bf5-ae74-4023-8c82-d79ce7db57dd/kube-multus/0.log"
Sep 16 02:58:02.368011 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:02.367983    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t4vxs_103d25bf-1d27-4cb9-be5c-cf6a8f8e691b/network-metrics-daemon/0.log"
Sep 16 02:58:02.383078 ip-10-0-143-48 kubenswrapper[2556]: I0916 02:58:02.383047    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t4vxs_103d25bf-1d27-4cb9-be5c-cf6a8f8e691b/kube-rbac-proxy/0.log"